Note: This is test shard 5 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250110 03:06:34.684908 14414 test_util.cc:274] Using random seed: 1642075613
I20250110 03:06:34.689817 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:06:34.689864 14414 ts_itest-base.cc:113] --------------
I20250110 03:06:34.689875 14414 ts_itest-base.cc:114] 6 tablet servers
I20250110 03:06:34.689888 14414 ts_itest-base.cc:115] 3 replicas per TS
I20250110 03:06:34.689898 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:06:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:06:34Z Disabled control of system clock
I20250110 03:06:34.697852 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:33919
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:33919
--location_mapping_cmd=/tmp/dist-test-taskWQrDfu/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20250110 03:06:34.771871 14426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:34.772081 14426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:34.772112 14426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:34.773409 14426 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:34.773458 14426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:06:34.773483 14426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:34.773500 14426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:06:34.773519 14426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:06:34.774966 14426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskWQrDfu/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:33919
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:34.775228 14426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:34.775450 14426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:34.779404 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:34.780470 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:34.781215 14434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:34.781972 14426 server_base.cc:1034] running on GCE node
I20250110 03:06:34.782171 14426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:34.782377 14426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:34.783497 14426 hybrid_clock.cc:648] HybridClock initialized: now 1736478394783476 us; error 41 us; skew 500 ppm
I20250110 03:06:34.790055 14426 webserver.cc:458] Webserver started at http://127.14.19.190:34875/ using document root <none> and password file <none>
I20250110 03:06:34.790246 14426 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:34.790288 14426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:34.790361 14426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:34.791344 14426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f89bff3d727e428ebd6894fe04bcefc8"
format_stamp: "Formatted at 2025-01-10 03:06:34 on dist-test-slave-nrvn"
I20250110 03:06:34.791611 14426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f89bff3d727e428ebd6894fe04bcefc8"
format_stamp: "Formatted at 2025-01-10 03:06:34 on dist-test-slave-nrvn"
I20250110 03:06:34.792716 14426 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:06:34.793398 14440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:34.793581 14426 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:34.793648 14426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f89bff3d727e428ebd6894fe04bcefc8"
format_stamp: "Formatted at 2025-01-10 03:06:34 on dist-test-slave-nrvn"
I20250110 03:06:34.793696 14426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:34.815804 14426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:34.816131 14426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:34.816236 14426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:34.819866 14426 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:33919
I20250110 03:06:34.819931 14492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:33919 every 8 connection(s)
I20250110 03:06:34.820222 14426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:06:34.820843 14493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:34.821816 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 14426
I20250110 03:06:34.821892 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:06:34.823552 14493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8: Bootstrap starting.
I20250110 03:06:34.824096 14493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:34.824333 14493 log.cc:826] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:34.824911 14493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8: No bootstrap required, opened a new log
I20250110 03:06:34.826604 14493 raft_consensus.cc:357] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89bff3d727e428ebd6894fe04bcefc8" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 33919 } }
I20250110 03:06:34.826732 14493 raft_consensus.cc:383] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:34.826761 14493 raft_consensus.cc:738] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f89bff3d727e428ebd6894fe04bcefc8, State: Initialized, Role: FOLLOWER
I20250110 03:06:34.826865 14493 consensus_queue.cc:260] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89bff3d727e428ebd6894fe04bcefc8" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 33919 } }
I20250110 03:06:34.826934 14493 raft_consensus.cc:397] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:06:34.826967 14493 raft_consensus.cc:491] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:06:34.827013 14493 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:34.827488 14493 raft_consensus.cc:513] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89bff3d727e428ebd6894fe04bcefc8" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 33919 } }
I20250110 03:06:34.827585 14493 leader_election.cc:304] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f89bff3d727e428ebd6894fe04bcefc8; no voters:
I20250110 03:06:34.827737 14493 leader_election.cc:290] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:06:34.827780 14498 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:34.827896 14498 raft_consensus.cc:695] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [term 1 LEADER]: Becoming Leader. State: Replica: f89bff3d727e428ebd6894fe04bcefc8, State: Running, Role: LEADER
I20250110 03:06:34.827984 14493 sys_catalog.cc:564] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:06:34.828066 14498 consensus_queue.cc:237] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89bff3d727e428ebd6894fe04bcefc8" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 33919 } }
I20250110 03:06:34.828399 14498 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f89bff3d727e428ebd6894fe04bcefc8. Latest consensus state: current_term: 1 leader_uuid: "f89bff3d727e428ebd6894fe04bcefc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89bff3d727e428ebd6894fe04bcefc8" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 33919 } } }
I20250110 03:06:34.828430 14500 sys_catalog.cc:455] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f89bff3d727e428ebd6894fe04bcefc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f89bff3d727e428ebd6894fe04bcefc8" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 33919 } } }
I20250110 03:06:34.828583 14500 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [sys.catalog]: This master's current role is: LEADER
I20250110 03:06:34.828912 14498 sys_catalog.cc:458] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8 [sys.catalog]: This master's current role is: LEADER
I20250110 03:06:34.828920 14506 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:06:34.829408 14506 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:06:34.831064 14506 catalog_manager.cc:1348] Generated new cluster ID: 4748a3054b1d4f94a2d697a064ed89c3
I20250110 03:06:34.831125 14506 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:06:34.842720 14506 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:06:34.843235 14506 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:06:34.850419 14506 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f89bff3d727e428ebd6894fe04bcefc8: Generated new TSK 0
I20250110 03:06:34.850622 14506 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:06:34.853701 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:34.930434 14517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:34.930673 14517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:34.930696 14517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:34.931993 14517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:34.932039 14517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:34.932063 14517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:06:34.933531 14517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--never_fsync=true
--heap_profile_path=/tmp/kudu.14517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:34.933741 14517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:34.934005 14517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:34.938211 14523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:34.939237 14522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:34.939992 14525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:34.940739 14517 server_base.cc:1034] running on GCE node
I20250110 03:06:34.940954 14517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:34.941241 14517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:34.942421 14517 hybrid_clock.cc:648] HybridClock initialized: now 1736478394942402 us; error 36 us; skew 500 ppm
I20250110 03:06:34.944073 14517 webserver.cc:458] Webserver started at http://127.14.19.129:34151/ using document root <none> and password file <none>
I20250110 03:06:34.944312 14517 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:34.944375 14517 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:34.944489 14517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:34.945710 14517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "97baa553877a44d2b1af2685ea7941a8"
format_stamp: "Formatted at 2025-01-10 03:06:34 on dist-test-slave-nrvn"
I20250110 03:06:34.946130 14517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "97baa553877a44d2b1af2685ea7941a8"
format_stamp: "Formatted at 2025-01-10 03:06:34 on dist-test-slave-nrvn"
I20250110 03:06:34.947846 14517 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20250110 03:06:34.948717 14531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:34.948895 14517 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:34.948984 14517 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "97baa553877a44d2b1af2685ea7941a8"
format_stamp: "Formatted at 2025-01-10 03:06:34 on dist-test-slave-nrvn"
I20250110 03:06:34.949067 14517 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:34.965651 14517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:34.965966 14517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:34.966123 14517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:34.966377 14517 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:34.966832 14517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:34.966887 14517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:34.966931 14517 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:34.966957 14517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:34.974486 14517 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:44589
I20250110 03:06:34.974529 14644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:44589 every 8 connection(s)
I20250110 03:06:34.974941 14517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:06:34.978310 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 14517
I20250110 03:06:34.978403 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:06:34.980391 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:34.980590 14645 heartbeater.cc:346] Connected to a master server at 127.14.19.190:33919
I20250110 03:06:34.980693 14645 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:34.980944 14645 heartbeater.cc:510] Master 127.14.19.190:33919 requested a full tablet report, sending...
I20250110 03:06:35.028002 14457 ts_manager.cc:194] Registered new tserver with Master: 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:35.028728 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:58543
W20250110 03:06:35.071739 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:35.071931 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:35.071957 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:35.073215 14648 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:35.073258 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:35.073293 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:06:35.074702 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--never_fsync=true
--heap_profile_path=/tmp/kudu.14648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:35.074929 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:35.075153 14648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:35.079221 14654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.080116 14655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.080806 14657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:35.081944 14648 server_base.cc:1034] running on GCE node
I20250110 03:06:35.082118 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:35.082319 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:35.083446 14648 hybrid_clock.cc:648] HybridClock initialized: now 1736478395083429 us; error 31 us; skew 500 ppm
I20250110 03:06:35.084700 14648 webserver.cc:458] Webserver started at http://127.14.19.130:33395/ using document root <none> and password file <none>
I20250110 03:06:35.084889 14648 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:35.084939 14648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:35.085064 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:35.085997 14648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c1502216bb0c469abd5491603f914b60"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.086315 14648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c1502216bb0c469abd5491603f914b60"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.087507 14648 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20250110 03:06:35.088099 14663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.088251 14648 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:35.088318 14648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c1502216bb0c469abd5491603f914b60"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.088380 14648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:35.096122 14648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:35.096346 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:35.096455 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:35.096626 14648 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:35.096977 14648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:35.097010 14648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.097047 14648 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:35.097069 14648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.102577 14648 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:45113
I20250110 03:06:35.102617 14776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:45113 every 8 connection(s)
I20250110 03:06:35.102895 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:06:35.104830 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 14648
I20250110 03:06:35.104907 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:06:35.106012 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:35.107805 14777 heartbeater.cc:346] Connected to a master server at 127.14.19.190:33919
I20250110 03:06:35.107901 14777 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:35.108101 14777 heartbeater.cc:510] Master 127.14.19.190:33919 requested a full tablet report, sending...
I20250110 03:06:35.149933 14457 ts_manager.cc:194] Registered new tserver with Master: c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:35.150457 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:54813
W20250110 03:06:35.199157 14780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:35.199400 14780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:35.199433 14780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:35.201498 14780 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:35.201570 14780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:35.201612 14780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:06:35.203871 14780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--never_fsync=true
--heap_profile_path=/tmp/kudu.14780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:35.204147 14780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:35.204416 14780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:35.209111 14789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.209856 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.210942 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:35.211773 14780 server_base.cc:1034] running on GCE node
I20250110 03:06:35.211954 14780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:35.212162 14780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:35.213286 14780 hybrid_clock.cc:648] HybridClock initialized: now 1736478395213269 us; error 31 us; skew 500 ppm
I20250110 03:06:35.214582 14780 webserver.cc:458] Webserver started at http://127.14.19.131:35301/ using document root <none> and password file <none>
I20250110 03:06:35.214792 14780 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:35.214844 14780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:35.214952 14780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:35.215842 14780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.216159 14780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.217311 14780 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20250110 03:06:35.217866 14795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.218039 14780 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:35.218132 14780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.218181 14780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:35.228003 14780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:35.228276 14780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:35.228384 14780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:35.228545 14780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:35.228812 14780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:35.228840 14780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.228868 14780 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:35.228884 14780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.234305 14780 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:33887
I20250110 03:06:35.234367 14908 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:33887 every 8 connection(s)
I20250110 03:06:35.234675 14780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:06:35.239166 14909 heartbeater.cc:346] Connected to a master server at 127.14.19.190:33919
I20250110 03:06:35.239254 14909 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:35.239421 14909 heartbeater.cc:510] Master 127.14.19.190:33919 requested a full tablet report, sending...
I20250110 03:06:35.240747 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 14780
I20250110 03:06:35.240834 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:06:35.242843 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:35.271747 14457 ts_manager.cc:194] Registered new tserver with Master: 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:35.272238 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:57857
W20250110 03:06:35.319463 14913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:35.319643 14913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:35.319675 14913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:35.320884 14913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:35.320928 14913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:35.320964 14913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:06:35.322425 14913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--never_fsync=true
--heap_profile_path=/tmp/kudu.14913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:35.322659 14913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:35.322891 14913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:35.327159 14918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.328114 14919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.329109 14921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:35.329855 14913 server_base.cc:1034] running on GCE node
I20250110 03:06:35.330060 14913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:35.330262 14913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:35.331393 14913 hybrid_clock.cc:648] HybridClock initialized: now 1736478395331375 us; error 32 us; skew 500 ppm
I20250110 03:06:35.332614 14913 webserver.cc:458] Webserver started at http://127.14.19.132:36297/ using document root <none> and password file <none>
I20250110 03:06:35.332851 14913 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:35.332902 14913 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:35.333000 14913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:35.333881 14913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.334195 14913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.335402 14913 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20250110 03:06:35.336071 14927 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.336221 14913 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:35.336294 14913 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.336355 14913 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:35.354022 14913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:35.354280 14913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:35.354393 14913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:35.354619 14913 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:35.354940 14913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:35.354979 14913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.355015 14913 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:35.355038 14913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.360482 14913 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:46299
I20250110 03:06:35.360543 15040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:46299 every 8 connection(s)
I20250110 03:06:35.360827 14913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:06:35.365435 15041 heartbeater.cc:346] Connected to a master server at 127.14.19.190:33919
I20250110 03:06:35.365523 15041 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:35.365725 15041 heartbeater.cc:510] Master 127.14.19.190:33919 requested a full tablet report, sending...
I20250110 03:06:35.366981 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 14913
I20250110 03:06:35.367061 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:06:35.368920 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:0
--local_ip_for_outbound_sockets=127.14.19.133
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:35.398885 14457 ts_manager.cc:194] Registered new tserver with Master: 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:35.399381 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:54937
W20250110 03:06:35.445801 15045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:35.446017 15045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:35.446050 15045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:35.447429 15045 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:35.447474 15045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:35.447497 15045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:06:35.448977 15045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--never_fsync=true
--heap_profile_path=/tmp/kudu.15045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:35.449203 15045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:35.449465 15045 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:35.454244 15050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.455128 15053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.455857 15051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:35.456593 15045 server_base.cc:1034] running on GCE node
I20250110 03:06:35.456751 15045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:35.456959 15045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:35.458086 15045 hybrid_clock.cc:648] HybridClock initialized: now 1736478395458070 us; error 31 us; skew 500 ppm
I20250110 03:06:35.459391 15045 webserver.cc:458] Webserver started at http://127.14.19.133:45499/ using document root <none> and password file <none>
I20250110 03:06:35.459589 15045 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:35.459641 15045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:35.459743 15045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:35.460631 15045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.460920 15045 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.462050 15045 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:06:35.462664 15059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.462821 15045 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:35.462879 15045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.462956 15045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:35.487208 15045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:35.487458 15045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:35.487565 15045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:35.487754 15045 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:35.488085 15045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:35.488116 15045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.488152 15045 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:35.488181 15045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.493595 15045 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:42181
I20250110 03:06:35.493669 15172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:42181 every 8 connection(s)
I20250110 03:06:35.493954 15045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:06:35.498842 15173 heartbeater.cc:346] Connected to a master server at 127.14.19.190:33919
I20250110 03:06:35.498937 15173 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:35.499135 15173 heartbeater.cc:510] Master 127.14.19.190:33919 requested a full tablet report, sending...
I20250110 03:06:35.503382 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 15045
I20250110 03:06:35.503477 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 03:06:35.504601 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:0
--local_ip_for_outbound_sockets=127.14.19.134
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--builtin_ntp_servers=127.14.19.148:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:35.544360 14457 ts_manager.cc:194] Registered new tserver with Master: c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:35.544883 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:48667
W20250110 03:06:35.594921 15177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:35.595090 15177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:35.595109 15177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:35.596385 15177 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:35.596431 15177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:35.596458 15177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:06:35.597916 15177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:33919
--never_fsync=true
--heap_profile_path=/tmp/kudu.15177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:35.598136 15177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:35.598366 15177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:35.603211 15182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.603283 15183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:35.604387 15185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:35.605151 15177 server_base.cc:1034] running on GCE node
I20250110 03:06:35.605319 15177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:35.605513 15177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:35.606591 15177 hybrid_clock.cc:648] HybridClock initialized: now 1736478395606571 us; error 34 us; skew 500 ppm
I20250110 03:06:35.607793 15177 webserver.cc:458] Webserver started at http://127.14.19.134:40923/ using document root <none> and password file <none>
I20250110 03:06:35.607976 15177 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:35.608021 15177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:35.608110 15177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:35.608991 15177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e7ff2606284a4b339d7d9b7e74256850"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.609287 15177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e7ff2606284a4b339d7d9b7e74256850"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.610432 15177 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20250110 03:06:35.611042 15191 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.611186 15177 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.611249 15177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e7ff2606284a4b339d7d9b7e74256850"
format_stamp: "Formatted at 2025-01-10 03:06:35 on dist-test-slave-nrvn"
I20250110 03:06:35.611300 15177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:35.620368 15177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:35.620594 15177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:35.620697 15177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:35.620862 15177 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:35.621132 15177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:35.621168 15177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.621201 15177 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:35.621227 15177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:35.626641 15177 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:37333
I20250110 03:06:35.626720 15304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:37333 every 8 connection(s)
I20250110 03:06:35.626993 15177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:06:35.629065 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 15177
I20250110 03:06:35.629155 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250110 03:06:35.632124 15305 heartbeater.cc:346] Connected to a master server at 127.14.19.190:33919
I20250110 03:06:35.632210 15305 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:35.632383 15305 heartbeater.cc:510] Master 127.14.19.190:33919 requested a full tablet report, sending...
I20250110 03:06:35.665036 14457 ts_manager.cc:194] Registered new tserver with Master: e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:35.665516 14457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:54039
I20250110 03:06:35.674445 14414 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250110 03:06:35.681591 14414 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250110 03:06:36.029814 14645 heartbeater.cc:502] Master 127.14.19.190:33919 was elected leader, sending a full tablet report...
I20250110 03:06:36.151577 14777 heartbeater.cc:502] Master 127.14.19.190:33919 was elected leader, sending a full tablet report...
I20250110 03:06:36.273391 14909 heartbeater.cc:502] Master 127.14.19.190:33919 was elected leader, sending a full tablet report...
I20250110 03:06:36.400278 15041 heartbeater.cc:502] Master 127.14.19.190:33919 was elected leader, sending a full tablet report...
I20250110 03:06:36.545919 15173 heartbeater.cc:502] Master 127.14.19.190:33919 was elected leader, sending a full tablet report...
I20250110 03:06:36.666406 15305 heartbeater.cc:502] Master 127.14.19.190:33919 was elected leader, sending a full tablet report...
I20250110 03:06:39.432603 14457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:39.441922 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 702128b35f2d4533944e12999ebac94f (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.442273 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702128b35f2d4533944e12999ebac94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.442232 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 702128b35f2d4533944e12999ebac94f (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.442517 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702128b35f2d4533944e12999ebac94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.444425 14579 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba26a65ff541999125575a675db2bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.444664 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ba26a65ff541999125575a675db2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.445472 14578 tablet_service.cc:1467] Processing CreateTablet for tablet a35848b976f74a81b13fd35452517de7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.445576 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35848b976f74a81b13fd35452517de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.445758 15239 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba26a65ff541999125575a675db2bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.446019 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ba26a65ff541999125575a675db2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.446823 15331 tablet_bootstrap.cc:492] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:39.447510 15331 tablet_bootstrap.cc:654] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.447810 15331 log.cc:826] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:39.448177 15330 tablet_bootstrap.cc:492] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:39.448802 15330 tablet_bootstrap.cc:654] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.448784 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 702128b35f2d4533944e12999ebac94f (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.449052 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 702128b35f2d4533944e12999ebac94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.449105 15330 log.cc:826] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:39.449872 15330 tablet_bootstrap.cc:492] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:39.449927 15330 ts_tablet_manager.cc:1397] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:39.451501 15332 tablet_bootstrap.cc:492] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:39.448804 15106 tablet_service.cc:1467] Processing CreateTablet for tablet a35848b976f74a81b13fd35452517de7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.452832 15330 raft_consensus.cc:357] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.452975 15330 raft_consensus.cc:383] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.453002 15330 raft_consensus.cc:738] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.452915 14843 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba26a65ff541999125575a675db2bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.453111 15330 consensus_queue.cc:260] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.453199 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4ba26a65ff541999125575a675db2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.453353 15330 ts_tablet_manager.cc:1428] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:39.453698 14842 tablet_service.cc:1467] Processing CreateTablet for tablet a35848b976f74a81b13fd35452517de7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1f4cbfa7c2ef4613bef82623a8ff4965]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.453868 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35848b976f74a81b13fd35452517de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.454298 15332 tablet_bootstrap.cc:654] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.454725 15332 log.cc:826] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:39.455329 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35848b976f74a81b13fd35452517de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.456391 15339 tablet_bootstrap.cc:492] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:39.456709 15334 tablet_bootstrap.cc:492] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:39.457001 15339 tablet_bootstrap.cc:654] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.457317 15339 log.cc:826] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:39.457842 15338 tablet_bootstrap.cc:492] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:39.458053 15331 tablet_bootstrap.cc:492] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:39.458133 15331 ts_tablet_manager.cc:1397] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20250110 03:06:39.458750 15338 tablet_bootstrap.cc:654] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.459075 15334 tablet_bootstrap.cc:654] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.459081 15338 log.cc:826] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:39.459472 15334 log.cc:826] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:39.459828 15332 tablet_bootstrap.cc:492] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:39.459923 15332 ts_tablet_manager.cc:1397] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20250110 03:06:39.460223 15338 tablet_bootstrap.cc:492] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:39.460283 15338 ts_tablet_manager.cc:1397] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:39.460685 15331 raft_consensus.cc:357] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.460795 15339 tablet_bootstrap.cc:492] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:39.460865 15331 raft_consensus.cc:383] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.460884 15339 ts_tablet_manager.cc:1397] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250110 03:06:39.460932 15331 raft_consensus.cc:738] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.461056 15331 consensus_queue.cc:260] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.461329 15331 ts_tablet_manager.cc:1428] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:39.462378 15332 raft_consensus.cc:357] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.462478 15338 raft_consensus.cc:357] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.462594 15338 raft_consensus.cc:383] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.462625 15338 raft_consensus.cc:738] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.462574 15332 raft_consensus.cc:383] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.462730 15332 raft_consensus.cc:738] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.462739 15338 consensus_queue.cc:260] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.463006 15338 ts_tablet_manager.cc:1428] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:39.463037 15334 tablet_bootstrap.cc:492] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:39.462944 15332 consensus_queue.cc:260] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.463093 15334 ts_tablet_manager.cc:1397] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250110 03:06:39.463356 15332 ts_tablet_manager.cc:1428] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20250110 03:06:39.463503 15332 tablet_bootstrap.cc:492] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:39.463573 15338 tablet_bootstrap.cc:492] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:39.463953 15332 tablet_bootstrap.cc:654] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.463994 15338 tablet_bootstrap.cc:654] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.464516 15338 tablet_bootstrap.cc:492] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:39.464570 15338 ts_tablet_manager.cc:1397] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:39.464716 15338 raft_consensus.cc:357] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.464792 15338 raft_consensus.cc:383] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.464814 15338 raft_consensus.cc:738] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.464895 15338 consensus_queue.cc:260] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.465003 15338 ts_tablet_manager.cc:1428] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.465269 15332 tablet_bootstrap.cc:492] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:39.465390 15332 ts_tablet_manager.cc:1397] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:06:39.465723 15332 raft_consensus.cc:357] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.465808 15332 raft_consensus.cc:383] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.465716 15334 raft_consensus.cc:357] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.465835 15332 raft_consensus.cc:738] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.465865 15334 raft_consensus.cc:383] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.465896 15334 raft_consensus.cc:738] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.465924 15332 consensus_queue.cc:260] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.466024 15332 ts_tablet_manager.cc:1428] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.466045 15334 consensus_queue.cc:260] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.466284 15334 ts_tablet_manager.cc:1428] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:39.467322 15339 raft_consensus.cc:357] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.467494 15339 raft_consensus.cc:383] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.467568 15339 raft_consensus.cc:738] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.467715 15339 consensus_queue.cc:260] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.468000 15339 ts_tablet_manager.cc:1428] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250110 03:06:39.468148 15339 tablet_bootstrap.cc:492] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:39.468765 15339 tablet_bootstrap.cc:654] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.469374 15339 tablet_bootstrap.cc:492] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:39.469417 15339 ts_tablet_manager.cc:1397] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:39.469554 15339 raft_consensus.cc:357] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.469625 15339 raft_consensus.cc:383] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.469655 15339 raft_consensus.cc:738] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.469698 15339 consensus_queue.cc:260] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.469785 15339 ts_tablet_manager.cc:1428] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20250110 03:06:39.478255 14646 tablet.cc:2367] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:06:39.487495 14910 tablet.cc:2367] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:06:39.497092 15174 tablet.cc:2367] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:39.555234 15344 raft_consensus.cc:491] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:39.555361 15344 raft_consensus.cc:513] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.555703 15344 leader_election.cc:290] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:39.559446 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35848b976f74a81b13fd35452517de7" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:39.559446 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35848b976f74a81b13fd35452517de7" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:39.559608 14599 raft_consensus.cc:2463] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:39.559608 14863 raft_consensus.cc:2463] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:39.559844 15060 leader_election.cc:304] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:39.559993 15344 raft_consensus.cc:2798] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:39.560052 15344 raft_consensus.cc:491] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:39.560072 15344 raft_consensus.cc:3054] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.560880 15344 raft_consensus.cc:513] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.560990 15344 leader_election.cc:290] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:39.561178 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35848b976f74a81b13fd35452517de7" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:39.561177 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35848b976f74a81b13fd35452517de7" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:39.561254 14599 raft_consensus.cc:3054] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.561259 14863 raft_consensus.cc:3054] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.561852 14599 raft_consensus.cc:2463] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:39.561899 14863 raft_consensus.cc:2463] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:39.562021 15060 leader_election.cc:304] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:39.562105 15344 raft_consensus.cc:2798] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:39.562264 15344 raft_consensus.cc:695] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 LEADER]: Becoming Leader. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Running, Role: LEADER
I20250110 03:06:39.562379 15344 consensus_queue.cc:237] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.563145 14457 catalog_manager.cc:5526] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:39.569520 15336 raft_consensus.cc:491] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:39.569622 15336 raft_consensus.cc:513] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.569880 15336 leader_election.cc:290] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:39.572654 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702128b35f2d4533944e12999ebac94f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:39.572695 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702128b35f2d4533944e12999ebac94f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:39.572795 15127 raft_consensus.cc:2463] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:39.572825 14995 raft_consensus.cc:2463] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:39.572983 14664 leader_election.cc:304] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:39.573098 15336 raft_consensus.cc:2798] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:39.573156 15336 raft_consensus.cc:491] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:39.573173 15336 raft_consensus.cc:3054] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.573761 15336 raft_consensus.cc:513] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.573873 15336 leader_election.cc:290] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:39.574026 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702128b35f2d4533944e12999ebac94f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:39.574100 14995 raft_consensus.cc:3054] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.574097 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702128b35f2d4533944e12999ebac94f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:39.574170 15127 raft_consensus.cc:3054] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.574811 14995 raft_consensus.cc:2463] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:39.574826 15127 raft_consensus.cc:2463] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:39.574985 14664 leader_election.cc:304] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:39.575083 15336 raft_consensus.cc:2798] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:39.575213 15336 raft_consensus.cc:695] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [term 1 LEADER]: Becoming Leader. State: Replica: c1502216bb0c469abd5491603f914b60, State: Running, Role: LEADER
I20250110 03:06:39.575289 15336 consensus_queue.cc:237] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:39.576006 14457 catalog_manager.cc:5526] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1502216bb0c469abd5491603f914b60 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "c1502216bb0c469abd5491603f914b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:39.606019 14778 tablet.cc:2367] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:06:39.613842 15042 tablet.cc:2367] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:06:39.629516 15306 tablet.cc:2367] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:39.673528 15348 raft_consensus.cc:491] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:39.673679 15348 raft_consensus.cc:513] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.674038 15348 leader_election.cc:290] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:39.676949 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba26a65ff541999125575a675db2bb" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:39.677068 14599 raft_consensus.cc:2463] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:39.677119 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba26a65ff541999125575a675db2bb" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:39.677227 14863 raft_consensus.cc:2463] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:39.677254 15192 leader_election.cc:304] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:39.677425 15348 raft_consensus.cc:2798] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:39.677475 15348 raft_consensus.cc:491] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:39.677502 15348 raft_consensus.cc:3054] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.678155 15348 raft_consensus.cc:513] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.678279 15348 leader_election.cc:290] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:39.678455 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba26a65ff541999125575a675db2bb" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:39.678459 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ba26a65ff541999125575a675db2bb" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:39.678530 14863 raft_consensus.cc:3054] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.678532 14599 raft_consensus.cc:3054] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.679095 14863 raft_consensus.cc:2463] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:39.679114 14599 raft_consensus.cc:2463] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:39.679262 15192 leader_election.cc:304] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:39.679353 15348 raft_consensus.cc:2798] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:39.679522 15348 raft_consensus.cc:695] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:39.679617 15348 consensus_queue.cc:237] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.680327 14457 catalog_manager.cc:5526] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:39.811257 15363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:39.811429 15363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:39.812655 15363 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250110 03:06:39.858099 14863 raft_consensus.cc:1270] T a35848b976f74a81b13fd35452517de7 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:39.858913 15344 consensus_queue.cc:1035] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:39.859346 14863 raft_consensus.cc:1270] T c4ba26a65ff541999125575a675db2bb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:39.859429 14599 raft_consensus.cc:1270] T a35848b976f74a81b13fd35452517de7 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:39.859561 15348 consensus_queue.cc:1035] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:39.859735 15353 consensus_queue.cc:1035] T a35848b976f74a81b13fd35452517de7 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:39.860473 15127 raft_consensus.cc:1270] T 702128b35f2d4533944e12999ebac94f P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:39.860528 14995 raft_consensus.cc:1270] T 702128b35f2d4533944e12999ebac94f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:39.860790 15357 consensus_queue.cc:1035] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:39.860982 15357 consensus_queue.cc:1035] T 702128b35f2d4533944e12999ebac94f P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:39.861203 14598 raft_consensus.cc:1270] T c4ba26a65ff541999125575a675db2bb P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:39.861419 15361 consensus_queue.cc:1035] T c4ba26a65ff541999125575a675db2bb P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:39.864287 15396 mvcc.cc:204] Tried to move back new op lower bound from 7112615525829779456 to 7112615525088034816. Current Snapshot: MvccSnapshot[applied={T|T < 7112615525821128704}]
I20250110 03:06:39.864459 15397 mvcc.cc:204] Tried to move back new op lower bound from 7112615525825654784 to 7112615524660744192. Current Snapshot: MvccSnapshot[applied={T|T < 7112615525825654784}]
I20250110 03:06:39.866117 15395 mvcc.cc:204] Tried to move back new op lower bound from 7112615525825654784 to 7112615524660744192. Current Snapshot: MvccSnapshot[applied={T|T < 7112615525825654784}]
I20250110 03:06:39.866564 15399 mvcc.cc:204] Tried to move back new op lower bound from 7112615525821128704 to 7112615525088034816. Current Snapshot: MvccSnapshot[applied={T|T < 7112615525821128704}]
I20250110 03:06:39.868016 15406 mvcc.cc:204] Tried to move back new op lower bound from 7112615525825654784 to 7112615524660744192. Current Snapshot: MvccSnapshot[applied={T|T < 7112615525825654784}]
INSERT report
rows total: 2000
time total: 35.4164 ms
time per row: 0.0177082 ms
I20250110 03:06:39.873708 15405 mvcc.cc:204] Tried to move back new op lower bound from 7112615525829779456 to 7112615525088034816. Current Snapshot: MvccSnapshot[applied={T|T < 7112615525829779456 or (T in {7112615525829779456})}]
I20250110 03:06:39.879366 14457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:39.880939 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfd7072076f419aa87d73e3f3b31abe (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.880987 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfd7072076f419aa87d73e3f3b31abe (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.881095 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dfd7072076f419aa87d73e3f3b31abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.881117 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dfd7072076f419aa87d73e3f3b31abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.881093 14843 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0669a30ff84567ac590e2811caa26e (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.881196 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b0669a30ff84567ac590e2811caa26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.882109 15331 tablet_bootstrap.cc:492] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:39.882364 15339 tablet_bootstrap.cc:492] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:39.882368 15334 tablet_bootstrap.cc:492] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:39.882395 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 31c76ba0cce9418fbe7c14f77778289f (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.882499 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c76ba0cce9418fbe7c14f77778289f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.882541 15331 tablet_bootstrap.cc:654] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.882788 15334 tablet_bootstrap.cc:654] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.882834 15339 tablet_bootstrap.cc:654] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.883111 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 6dfd7072076f419aa87d73e3f3b31abe (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.883230 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dfd7072076f419aa87d73e3f3b31abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.883473 15339 tablet_bootstrap.cc:492] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:39.883513 15339 ts_tablet_manager.cc:1397] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:39.883559 15331 tablet_bootstrap.cc:492] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:39.883615 15331 ts_tablet_manager.cc:1397] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:39.883657 15339 raft_consensus.cc:357] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.883718 15339 raft_consensus.cc:383] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.883738 15339 raft_consensus.cc:738] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.883786 15339 consensus_queue.cc:260] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.883778 15331 raft_consensus.cc:357] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.883844 15331 raft_consensus.cc:383] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.883869 15331 raft_consensus.cc:738] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.883895 15339 ts_tablet_manager.cc:1428] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.883929 15331 consensus_queue.cc:260] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.884027 15331 ts_tablet_manager.cc:1428] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:39.884430 15330 tablet_bootstrap.cc:492] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:39.884471 14710 tablet_service.cc:1467] Processing CreateTablet for tablet 31c76ba0cce9418fbe7c14f77778289f (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.884552 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c76ba0cce9418fbe7c14f77778289f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.884682 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0669a30ff84567ac590e2811caa26e (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.884761 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0669a30ff84567ac590e2811caa26e (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.884799 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b0669a30ff84567ac590e2811caa26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.884847 15330 tablet_bootstrap.cc:654] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.884865 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b0669a30ff84567ac590e2811caa26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.886071 15332 tablet_bootstrap.cc:492] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:39.886094 15338 tablet_bootstrap.cc:492] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:39.886140 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 31c76ba0cce9418fbe7c14f77778289f (DEFAULT_TABLE table=rebalance_test_table_1 [id=61c852a1037144409eab935b8237a598]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:39.886235 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31c76ba0cce9418fbe7c14f77778289f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:39.886468 15338 tablet_bootstrap.cc:654] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.886467 15332 tablet_bootstrap.cc:654] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.887140 15332 tablet_bootstrap.cc:492] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:39.887141 15330 tablet_bootstrap.cc:492] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:39.887187 15332 ts_tablet_manager.cc:1397] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:39.887188 15330 ts_tablet_manager.cc:1397] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:39.887336 15332 raft_consensus.cc:357] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.887346 15330 raft_consensus.cc:357] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.887403 15332 raft_consensus.cc:383] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.887419 15330 raft_consensus.cc:383] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.887427 15332 raft_consensus.cc:738] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.887441 15330 raft_consensus.cc:738] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.887478 15338 tablet_bootstrap.cc:492] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:39.887473 15332 consensus_queue.cc:260] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.887482 15330 consensus_queue.cc:260] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.887516 15338 ts_tablet_manager.cc:1397] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:39.887571 15332 ts_tablet_manager.cc:1428] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.887574 15330 ts_tablet_manager.cc:1428] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:39.887631 15330 tablet_bootstrap.cc:492] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:39.887666 15338 raft_consensus.cc:357] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.887724 15331 tablet_bootstrap.cc:492] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:39.887733 15338 raft_consensus.cc:383] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.887755 15338 raft_consensus.cc:738] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.887792 15338 consensus_queue.cc:260] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.887881 15338 ts_tablet_manager.cc:1428] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.887972 15334 tablet_bootstrap.cc:492] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:39.887998 15330 tablet_bootstrap.cc:654] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.888020 15334 ts_tablet_manager.cc:1397] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20250110 03:06:39.888137 15331 tablet_bootstrap.cc:654] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.888160 15334 raft_consensus.cc:357] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.888227 15334 raft_consensus.cc:383] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.888250 15334 raft_consensus.cc:738] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.888311 15334 consensus_queue.cc:260] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.888399 15334 ts_tablet_manager.cc:1428] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:39.888479 15334 tablet_bootstrap.cc:492] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:39.888734 15330 tablet_bootstrap.cc:492] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:39.888782 15330 ts_tablet_manager.cc:1397] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:39.888916 15330 raft_consensus.cc:357] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.888978 15330 raft_consensus.cc:383] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.889003 15330 raft_consensus.cc:738] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.889048 15330 consensus_queue.cc:260] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.889104 15334 tablet_bootstrap.cc:654] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:39.889178 15330 ts_tablet_manager.cc:1428] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.889222 15331 tablet_bootstrap.cc:492] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:39.889259 15331 ts_tablet_manager.cc:1397] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:39.889403 15331 raft_consensus.cc:357] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.889459 15331 raft_consensus.cc:383] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.889478 15331 raft_consensus.cc:738] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.889531 15331 consensus_queue.cc:260] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.889657 15331 ts_tablet_manager.cc:1428] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.889793 15334 tablet_bootstrap.cc:492] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:39.889834 15334 ts_tablet_manager.cc:1397] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:39.889971 15334 raft_consensus.cc:357] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.890025 15334 raft_consensus.cc:383] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:39.890043 15334 raft_consensus.cc:738] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:39.890092 15334 consensus_queue.cc:260] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.890172 15334 ts_tablet_manager.cc:1428] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:39.945736 15336 raft_consensus.cc:491] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:39.945828 15336 raft_consensus.cc:513] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.946040 15336 leader_election.cc:290] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:39.946314 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76ba0cce9418fbe7c14f77778289f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:39.946430 14995 raft_consensus.cc:2463] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:39.946679 14664 leader_election.cc:304] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:39.946775 15336 raft_consensus.cc:2798] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:39.946815 15336 raft_consensus.cc:491] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:39.946836 15336 raft_consensus.cc:3054] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.947460 15336 raft_consensus.cc:513] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.947607 15336 leader_election.cc:290] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Requested vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:39.947827 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76ba0cce9418fbe7c14f77778289f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:39.947921 14995 raft_consensus.cc:3054] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.948647 14995 raft_consensus.cc:2463] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:39.948839 14664 leader_election.cc:304] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:39.948945 15336 raft_consensus.cc:2798] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:39.948896 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76ba0cce9418fbe7c14f77778289f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:39.949010 15336 raft_consensus.cc:695] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [term 1 LEADER]: Becoming Leader. State: Replica: c1502216bb0c469abd5491603f914b60, State: Running, Role: LEADER
I20250110 03:06:39.948899 15259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31c76ba0cce9418fbe7c14f77778289f" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:39.949052 15258 raft_consensus.cc:3054] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.949066 15336 consensus_queue.cc:237] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:39.949580 15348 raft_consensus.cc:491] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:39.949642 15348 raft_consensus.cc:513] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.949807 15348 leader_election.cc:290] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:39.949939 15258 raft_consensus.cc:2463] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:39.950469 15413 raft_consensus.cc:491] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:39.950471 14457 catalog_manager.cc:5526] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1502216bb0c469abd5491603f914b60 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "c1502216bb0c469abd5491603f914b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:39.950574 15413 raft_consensus.cc:513] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.950722 15413 leader_election.cc:290] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:39.951787 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0669a30ff84567ac590e2811caa26e" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:39.951884 14599 raft_consensus.cc:2463] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:39.952095 15060 leader_election.cc:304] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:39.952196 15413 raft_consensus.cc:2798] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:39.952246 15413 raft_consensus.cc:491] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:39.952270 15413 raft_consensus.cc:3054] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.952272 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0669a30ff84567ac590e2811caa26e" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:39.952358 14862 raft_consensus.cc:2463] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:39.953111 15413 raft_consensus.cc:513] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.953148 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd7072076f419aa87d73e3f3b31abe" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:39.953244 15413 leader_election.cc:290] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:39.953243 14731 raft_consensus.cc:2463] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:39.953423 15195 leader_election.cc:304] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:39.953541 15348 raft_consensus.cc:2798] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:39.953572 15348 raft_consensus.cc:491] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:39.953539 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0669a30ff84567ac590e2811caa26e" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:39.953594 15348 raft_consensus.cc:3054] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.953568 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd7072076f419aa87d73e3f3b31abe" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:39.953625 14995 raft_consensus.cc:2463] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:39.953620 14599 raft_consensus.cc:3054] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.954257 15348 raft_consensus.cc:513] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.954378 14599 raft_consensus.cc:2463] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:39.954398 15348 leader_election.cc:290] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:39.954519 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd7072076f419aa87d73e3f3b31abe" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:39.954605 14995 raft_consensus.cc:3054] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.954592 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dfd7072076f419aa87d73e3f3b31abe" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:39.954649 14731 raft_consensus.cc:3054] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.954653 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0669a30ff84567ac590e2811caa26e" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:39.954725 14862 raft_consensus.cc:3054] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:39.955279 14862 raft_consensus.cc:2463] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:39.955282 14995 raft_consensus.cc:2463] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:39.955360 14731 raft_consensus.cc:2463] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:39.955377 15060 leader_election.cc:304] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:39.955476 15413 raft_consensus.cc:2798] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:39.955495 15195 leader_election.cc:304] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:39.955519 15413 raft_consensus.cc:695] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 LEADER]: Becoming Leader. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Running, Role: LEADER
I20250110 03:06:39.955566 15413 consensus_queue.cc:237] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:39.955838 15348 raft_consensus.cc:2798] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:39.955924 15348 raft_consensus.cc:695] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:39.955976 15348 consensus_queue.cc:237] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:39.956168 14457 catalog_manager.cc:5526] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:39.956528 14455 catalog_manager.cc:5526] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:40.043278 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:40.043476 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:40.044809 15442 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250110 03:06:40.074297 15258 raft_consensus.cc:1270] T 31c76ba0cce9418fbe7c14f77778289f P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.074585 15357 consensus_queue.cc:1035] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.074987 14995 raft_consensus.cc:1270] T 31c76ba0cce9418fbe7c14f77778289f P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.075299 15357 consensus_queue.cc:1035] T 31c76ba0cce9418fbe7c14f77778289f P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.075407 14862 raft_consensus.cc:1270] T 7b0669a30ff84567ac590e2811caa26e P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.075733 14731 raft_consensus.cc:1270] T 6dfd7072076f419aa87d73e3f3b31abe P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.075879 15353 consensus_queue.cc:1035] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.076398 14599 raft_consensus.cc:1270] T 7b0669a30ff84567ac590e2811caa26e P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.076560 14994 raft_consensus.cc:1270] T 6dfd7072076f419aa87d73e3f3b31abe P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.076822 15353 consensus_queue.cc:1035] T 7b0669a30ff84567ac590e2811caa26e P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.076825 15348 consensus_queue.cc:1035] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.077544 15348 consensus_queue.cc:1035] T 6dfd7072076f419aa87d73e3f3b31abe P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 24.9679 ms
time per row: 0.0124839 ms
I20250110 03:06:40.097163 14454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:40.098906 14843 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3b6ce3849f44549336863083d51d97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.098971 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 698676092ce04c6697e5e06a94f2ef57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.099088 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698676092ce04c6697e5e06a94f2ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.099123 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 9435af43b1d3449cae7e07a89152edfb (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.099197 14710 tablet_service.cc:1467] Processing CreateTablet for tablet 698676092ce04c6697e5e06a94f2ef57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.099256 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9435af43b1d3449cae7e07a89152edfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.099300 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698676092ce04c6697e5e06a94f2ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.099275 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3b6ce3849f44549336863083d51d97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.099380 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3b6ce3849f44549336863083d51d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.100404 15331 tablet_bootstrap.cc:492] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:40.100450 15330 tablet_bootstrap.cc:492] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:40.100574 15332 tablet_bootstrap.cc:492] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:40.100605 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 9435af43b1d3449cae7e07a89152edfb (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.100701 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9435af43b1d3449cae7e07a89152edfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.100900 15330 tablet_bootstrap.cc:654] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.101004 15332 tablet_bootstrap.cc:654] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.101066 15331 tablet_bootstrap.cc:654] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.101450 15339 tablet_bootstrap.cc:492] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:40.101581 15332 tablet_bootstrap.cc:492] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:40.101626 15332 ts_tablet_manager.cc:1397] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.101661 15331 tablet_bootstrap.cc:492] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:40.101706 15331 ts_tablet_manager.cc:1397] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:40.101787 15332 raft_consensus.cc:357] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.101833 15339 tablet_bootstrap.cc:654] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.101852 15332 raft_consensus.cc:383] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.101874 15332 raft_consensus.cc:738] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.101831 15331 raft_consensus.cc:357] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.101902 15331 raft_consensus.cc:383] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.101923 15331 raft_consensus.cc:738] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.101922 15332 consensus_queue.cc:260] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.101964 15331 consensus_queue.cc:260] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.102015 15332 ts_tablet_manager.cc:1428] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:40.102046 15331 ts_tablet_manager.cc:1428] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.102031 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 9e3b6ce3849f44549336863083d51d97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.102070 15332 tablet_bootstrap.cc:492] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:40.102142 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3b6ce3849f44549336863083d51d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.102383 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 698676092ce04c6697e5e06a94f2ef57 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.102450 15332 tablet_bootstrap.cc:654] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.102485 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698676092ce04c6697e5e06a94f2ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.102854 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3b6ce3849f44549336863083d51d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.103456 15338 tablet_bootstrap.cc:492] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:40.103813 15338 tablet_bootstrap.cc:654] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.103814 15334 tablet_bootstrap.cc:492] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:40.104269 15334 tablet_bootstrap.cc:654] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.104719 15338 tablet_bootstrap.cc:492] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:40.104764 15338 ts_tablet_manager.cc:1397] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:40.104856 15334 tablet_bootstrap.cc:492] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:40.104907 15334 ts_tablet_manager.cc:1397] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.104889 15338 raft_consensus.cc:357] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.104947 15338 raft_consensus.cc:383] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.104967 15338 raft_consensus.cc:738] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.105007 15338 consensus_queue.cc:260] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.105042 15334 raft_consensus.cc:357] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.105098 15334 raft_consensus.cc:383] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.105096 15338 ts_tablet_manager.cc:1428] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.105118 15334 raft_consensus.cc:738] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.105165 15334 consensus_queue.cc:260] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.105250 15334 ts_tablet_manager.cc:1428] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.106839 15339 tablet_bootstrap.cc:492] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:40.106889 15339 ts_tablet_manager.cc:1397] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250110 03:06:40.107041 15339 raft_consensus.cc:357] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.107112 15339 raft_consensus.cc:383] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.107146 15339 raft_consensus.cc:738] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.107201 15339 consensus_queue.cc:260] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.107298 15339 ts_tablet_manager.cc:1428] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.107357 15339 tablet_bootstrap.cc:492] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:40.107725 15339 tablet_bootstrap.cc:654] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.107982 15332 tablet_bootstrap.cc:492] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:40.108039 15332 ts_tablet_manager.cc:1397] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20250110 03:06:40.108201 15332 raft_consensus.cc:357] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.108271 15332 raft_consensus.cc:383] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.108294 15332 raft_consensus.cc:738] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.108340 15332 consensus_queue.cc:260] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.108355 15339 tablet_bootstrap.cc:492] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:40.108398 15339 ts_tablet_manager.cc:1397] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.108433 15332 ts_tablet_manager.cc:1428] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.108485 15330 tablet_bootstrap.cc:492] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:40.108528 15330 ts_tablet_manager.cc:1397] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250110 03:06:40.108539 15339 raft_consensus.cc:357] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.108599 15339 raft_consensus.cc:383] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.108618 15339 raft_consensus.cc:738] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.108641 15330 raft_consensus.cc:357] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.108676 15339 consensus_queue.cc:260] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.108711 15330 raft_consensus.cc:383] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.108733 15330 raft_consensus.cc:738] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.108722 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 9435af43b1d3449cae7e07a89152edfb (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3405b7401bc4ea391c051e8b78319e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.108790 15339 ts_tablet_manager.cc:1428] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.108800 15330 consensus_queue.cc:260] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.108845 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9435af43b1d3449cae7e07a89152edfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.108906 15330 ts_tablet_manager.cc:1428] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:40.109848 15338 tablet_bootstrap.cc:492] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:40.110285 15338 tablet_bootstrap.cc:654] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.111197 15338 tablet_bootstrap.cc:492] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:40.111238 15338 ts_tablet_manager.cc:1397] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:40.111342 15338 raft_consensus.cc:357] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.111387 15338 raft_consensus.cc:383] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.111402 15338 raft_consensus.cc:738] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.111438 15338 consensus_queue.cc:260] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.111544 15338 ts_tablet_manager.cc:1428] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.220242 15357 raft_consensus.cc:491] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.220355 15357 raft_consensus.cc:513] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.220516 15357 leader_election.cc:290] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:40.220794 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698676092ce04c6697e5e06a94f2ef57" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:40.220808 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698676092ce04c6697e5e06a94f2ef57" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:40.220901 15258 raft_consensus.cc:2463] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:40.220943 14994 raft_consensus.cc:2463] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:40.221056 14667 leader_election.cc:304] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:40.221187 15357 raft_consensus.cc:2798] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.221239 15357 raft_consensus.cc:491] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.221262 15357 raft_consensus.cc:3054] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.221973 15357 raft_consensus.cc:513] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.222129 15357 leader_election.cc:290] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:40.222337 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698676092ce04c6697e5e06a94f2ef57" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:40.222326 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698676092ce04c6697e5e06a94f2ef57" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:40.222409 14994 raft_consensus.cc:3054] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.222409 15258 raft_consensus.cc:3054] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.223220 14994 raft_consensus.cc:2463] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:40.223222 15258 raft_consensus.cc:2463] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:40.223389 14667 leader_election.cc:304] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:40.223515 15357 raft_consensus.cc:2798] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.223575 15357 raft_consensus.cc:695] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [term 1 LEADER]: Becoming Leader. State: Replica: c1502216bb0c469abd5491603f914b60, State: Running, Role: LEADER
I20250110 03:06:40.223631 15357 consensus_queue.cc:237] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:40.224210 14454 catalog_manager.cc:5526] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1502216bb0c469abd5491603f914b60 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "c1502216bb0c469abd5491603f914b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:40.230636 15398 raft_consensus.cc:491] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.230716 15398 raft_consensus.cc:513] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.231017 15398 leader_election.cc:290] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:40.233551 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435af43b1d3449cae7e07a89152edfb" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:40.233654 14599 raft_consensus.cc:2463] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:40.233824 14796 leader_election.cc:304] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:40.233945 15398 raft_consensus.cc:2798] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.233995 15398 raft_consensus.cc:491] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.234009 15398 raft_consensus.cc:3054] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.234167 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435af43b1d3449cae7e07a89152edfb" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:40.234246 15127 raft_consensus.cc:2463] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:40.234776 15398 raft_consensus.cc:513] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.234908 15398 leader_election.cc:290] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:40.235092 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435af43b1d3449cae7e07a89152edfb" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:40.235107 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9435af43b1d3449cae7e07a89152edfb" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:40.235172 14599 raft_consensus.cc:3054] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.235179 15127 raft_consensus.cc:3054] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.235769 15127 raft_consensus.cc:2463] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:40.235911 14599 raft_consensus.cc:2463] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:40.236027 14796 leader_election.cc:304] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:40.236110 15398 raft_consensus.cc:2798] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.236202 15398 raft_consensus.cc:695] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Running, Role: LEADER
I20250110 03:06:40.236277 15398 consensus_queue.cc:237] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.237017 14454 catalog_manager.cc:5526] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131). New cstate: current_term: 1 leader_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:40.334671 15353 raft_consensus.cc:491] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.334774 15353 raft_consensus.cc:513] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.334919 15353 leader_election.cc:290] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:40.335115 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3b6ce3849f44549336863083d51d97" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:40.335119 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3b6ce3849f44549336863083d51d97" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:40.335199 14599 raft_consensus.cc:2463] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:40.335222 14862 raft_consensus.cc:2463] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:40.335348 15060 leader_election.cc:304] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:40.335476 15353 raft_consensus.cc:2798] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.335521 15353 raft_consensus.cc:491] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.335536 15353 raft_consensus.cc:3054] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.336200 15353 raft_consensus.cc:513] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.336315 15353 leader_election.cc:290] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:40.336490 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3b6ce3849f44549336863083d51d97" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:40.336491 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e3b6ce3849f44549336863083d51d97" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:40.336571 14862 raft_consensus.cc:3054] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.336571 14599 raft_consensus.cc:3054] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.337273 14862 raft_consensus.cc:2463] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:40.337275 14599 raft_consensus.cc:2463] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:40.337455 15060 leader_election.cc:304] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:40.337541 15353 raft_consensus.cc:2798] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.337600 15353 raft_consensus.cc:695] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 LEADER]: Becoming Leader. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Running, Role: LEADER
I20250110 03:06:40.337654 15353 consensus_queue.cc:237] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.338244 14454 catalog_manager.cc:5526] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:40.466517 15493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:40.466727 15493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:40.468017 15493 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20250110 03:06:40.502445 14862 raft_consensus.cc:1270] T 9e3b6ce3849f44549336863083d51d97 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.502491 14599 raft_consensus.cc:1270] T 9e3b6ce3849f44549336863083d51d97 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.502813 15344 consensus_queue.cc:1035] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.503147 15344 consensus_queue.cc:1035] T 9e3b6ce3849f44549336863083d51d97 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.504702 15127 raft_consensus.cc:1270] T 9435af43b1d3449cae7e07a89152edfb P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.504882 14598 raft_consensus.cc:1270] T 9435af43b1d3449cae7e07a89152edfb P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.505085 15258 raft_consensus.cc:1270] T 698676092ce04c6697e5e06a94f2ef57 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.505473 14994 raft_consensus.cc:1270] T 698676092ce04c6697e5e06a94f2ef57 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.505539 15357 consensus_queue.cc:1035] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.506181 15349 consensus_queue.cc:1035] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.506297 15398 consensus_queue.cc:1035] T 9435af43b1d3449cae7e07a89152edfb P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.507185 15357 consensus_queue.cc:1035] T 698676092ce04c6697e5e06a94f2ef57 P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 32.6848 ms
time per row: 0.0163424 ms
I20250110 03:06:40.526984 14454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:40.528622 14975 tablet_service.cc:1467] Processing CreateTablet for tablet c86f946ddcd94c40ac5f6f3db19e6bdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.528614 15238 tablet_service.cc:1467] Processing CreateTablet for tablet c86f946ddcd94c40ac5f6f3db19e6bdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.528614 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7dc51d764044e8a6df257169f4d175 (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.528702 14843 tablet_service.cc:1467] Processing CreateTablet for tablet d7a3695e3b2849fdb1f6fa2e1fcfbc64 (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.528793 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86f946ddcd94c40ac5f6f3db19e6bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.528793 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86f946ddcd94c40ac5f6f3db19e6bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.528813 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a3695e3b2849fdb1f6fa2e1fcfbc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.529083 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7dc51d764044e8a6df257169f4d175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.530120 15334 tablet_bootstrap.cc:492] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:40.530117 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7dc51d764044e8a6df257169f4d175 (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.530213 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7dc51d764044e8a6df257169f4d175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.530211 15339 tablet_bootstrap.cc:492] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:40.530314 15331 tablet_bootstrap.cc:492] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:40.530508 15334 tablet_bootstrap.cc:654] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.530642 15339 tablet_bootstrap.cc:654] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.530817 15331 tablet_bootstrap.cc:654] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.531661 14578 tablet_service.cc:1467] Processing CreateTablet for tablet d7a3695e3b2849fdb1f6fa2e1fcfbc64 (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.531782 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a3695e3b2849fdb1f6fa2e1fcfbc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.531751 15106 tablet_service.cc:1467] Processing CreateTablet for tablet d7a3695e3b2849fdb1f6fa2e1fcfbc64 (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.531816 14710 tablet_service.cc:1467] Processing CreateTablet for tablet c86f946ddcd94c40ac5f6f3db19e6bdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.531883 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7a3695e3b2849fdb1f6fa2e1fcfbc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.531929 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c86f946ddcd94c40ac5f6f3db19e6bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.532909 15331 tablet_bootstrap.cc:492] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:40.532914 15339 tablet_bootstrap.cc:492] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:40.532953 15331 ts_tablet_manager.cc:1397] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:40.532953 15339 ts_tablet_manager.cc:1397] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:40.533092 15338 tablet_bootstrap.cc:492] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:40.533083 15331 raft_consensus.cc:357] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.533118 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7dc51d764044e8a6df257169f4d175 (DEFAULT_TABLE table=rebalance_test_table_3 [id=718e316f7102433b90e19861a286b363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.533144 15331 raft_consensus.cc:383] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.533102 15339 raft_consensus.cc:357] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.533166 15331 raft_consensus.cc:738] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.533169 15339 raft_consensus.cc:383] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.533191 15339 raft_consensus.cc:738] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.533200 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c7dc51d764044e8a6df257169f4d175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.533201 15331 consensus_queue.cc:260] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.533231 15339 consensus_queue.cc:260] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.533316 15331 ts_tablet_manager.cc:1428] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.533321 15339 ts_tablet_manager.cc:1428] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:40.533380 15331 tablet_bootstrap.cc:492] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:40.533488 15338 tablet_bootstrap.cc:654] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.533572 15334 tablet_bootstrap.cc:492] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:40.533618 15334 ts_tablet_manager.cc:1397] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:06:40.533797 15331 tablet_bootstrap.cc:654] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.533784 15334 raft_consensus.cc:357] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.533849 15334 raft_consensus.cc:383] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.533881 15334 raft_consensus.cc:738] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.533931 15334 consensus_queue.cc:260] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.534037 15334 ts_tablet_manager.cc:1428] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.534113 15334 tablet_bootstrap.cc:492] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:40.534191 15332 tablet_bootstrap.cc:492] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:40.534528 15334 tablet_bootstrap.cc:654] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.534642 15332 tablet_bootstrap.cc:654] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.534677 15331 tablet_bootstrap.cc:492] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:40.534714 15331 ts_tablet_manager.cc:1397] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.534869 15331 raft_consensus.cc:357] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.534916 15330 tablet_bootstrap.cc:492] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:40.534940 15331 raft_consensus.cc:383] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.534965 15331 raft_consensus.cc:738] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.535007 15331 consensus_queue.cc:260] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.535156 15331 ts_tablet_manager.cc:1428] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:40.535310 15334 tablet_bootstrap.cc:492] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:40.535346 15334 ts_tablet_manager.cc:1397] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.535370 15330 tablet_bootstrap.cc:654] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.535462 15334 raft_consensus.cc:357] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.535518 15334 raft_consensus.cc:383] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.535535 15334 raft_consensus.cc:738] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.535588 15334 consensus_queue.cc:260] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.535669 15334 ts_tablet_manager.cc:1428] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.536787 15330 tablet_bootstrap.cc:492] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:40.536834 15330 ts_tablet_manager.cc:1397] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:40.536970 15332 tablet_bootstrap.cc:492] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:40.536974 15330 raft_consensus.cc:357] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.537032 15330 raft_consensus.cc:383] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.537030 15332 ts_tablet_manager.cc:1397] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:40.537067 15330 raft_consensus.cc:738] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.537113 15330 consensus_queue.cc:260] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.537217 15330 ts_tablet_manager.cc:1428] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:40.537217 15332 raft_consensus.cc:357] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.537279 15330 tablet_bootstrap.cc:492] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:40.537287 15332 raft_consensus.cc:383] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.537307 15332 raft_consensus.cc:738] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.537353 15332 consensus_queue.cc:260] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.537437 15332 ts_tablet_manager.cc:1428] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:40.537681 15330 tablet_bootstrap.cc:654] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.537870 15338 tablet_bootstrap.cc:492] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:40.537910 15338 ts_tablet_manager.cc:1397] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250110 03:06:40.538053 15338 raft_consensus.cc:357] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.538148 15338 raft_consensus.cc:383] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.538168 15338 raft_consensus.cc:738] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.538213 15338 consensus_queue.cc:260] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.538277 15330 tablet_bootstrap.cc:492] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:40.538282 15338 ts_tablet_manager.cc:1428] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.538317 15330 ts_tablet_manager.cc:1397] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:40.538450 15330 raft_consensus.cc:357] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.538525 15330 raft_consensus.cc:383] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.538591 15330 raft_consensus.cc:738] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.538635 15330 consensus_queue.cc:260] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.538719 15330 ts_tablet_manager.cc:1428] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:40.556135 15530 raft_consensus.cc:491] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.556241 15530 raft_consensus.cc:513] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.556418 15530 leader_election.cc:290] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:40.556623 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f946ddcd94c40ac5f6f3db19e6bdc" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:40.556670 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f946ddcd94c40ac5f6f3db19e6bdc" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:40.556726 14994 raft_consensus.cc:2463] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:40.556748 15258 raft_consensus.cc:2463] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:40.556908 14667 leader_election.cc:304] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:40.557008 15530 raft_consensus.cc:2798] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.557061 15530 raft_consensus.cc:491] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.557106 15530 raft_consensus.cc:3054] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.557803 15530 raft_consensus.cc:513] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.557924 15530 leader_election.cc:290] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:40.558109 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f946ddcd94c40ac5f6f3db19e6bdc" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:40.558099 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86f946ddcd94c40ac5f6f3db19e6bdc" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:40.558177 14994 raft_consensus.cc:3054] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.558177 15258 raft_consensus.cc:3054] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.558936 15258 raft_consensus.cc:2463] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:40.558936 14994 raft_consensus.cc:2463] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:40.559160 14667 leader_election.cc:304] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:40.559278 15530 raft_consensus.cc:2798] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.559340 15530 raft_consensus.cc:695] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [term 1 LEADER]: Becoming Leader. State: Replica: c1502216bb0c469abd5491603f914b60, State: Running, Role: LEADER
I20250110 03:06:40.559386 15530 consensus_queue.cc:237] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.560058 14454 catalog_manager.cc:5526] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1502216bb0c469abd5491603f914b60 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "c1502216bb0c469abd5491603f914b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:40.594892 15361 raft_consensus.cc:491] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.594995 15361 raft_consensus.cc:513] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.595176 15361 leader_election.cc:290] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:40.595391 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7dc51d764044e8a6df257169f4d175" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:40.595405 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7dc51d764044e8a6df257169f4d175" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:40.595485 14731 raft_consensus.cc:2463] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:40.595520 14994 raft_consensus.cc:2463] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:40.595630 15195 leader_election.cc:304] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:40.595731 15361 raft_consensus.cc:2798] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.595795 15361 raft_consensus.cc:491] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.595829 15361 raft_consensus.cc:3054] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.596504 15361 raft_consensus.cc:513] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.596643 15361 leader_election.cc:290] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:40.596791 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7dc51d764044e8a6df257169f4d175" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:40.596798 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7dc51d764044e8a6df257169f4d175" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:40.596861 14731 raft_consensus.cc:3054] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.596875 14994 raft_consensus.cc:3054] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.597648 14731 raft_consensus.cc:2463] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:40.597666 14994 raft_consensus.cc:2463] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:40.597879 15192 leader_election.cc:304] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:40.597994 15361 raft_consensus.cc:2798] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.598156 15361 raft_consensus.cc:695] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:40.598217 15361 consensus_queue.cc:237] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:40.598876 14454 catalog_manager.cc:5526] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:40.665854 15345 raft_consensus.cc:491] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.665961 15345 raft_consensus.cc:513] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.666338 15345 leader_election.cc:290] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:40.669045 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a3695e3b2849fdb1f6fa2e1fcfbc64" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:40.669102 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a3695e3b2849fdb1f6fa2e1fcfbc64" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:40.669165 15127 raft_consensus.cc:2463] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:40.669184 14862 raft_consensus.cc:2463] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:40.669363 14532 leader_election.cc:304] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:40.669495 15345 raft_consensus.cc:2798] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.669545 15345 raft_consensus.cc:491] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.669561 15345 raft_consensus.cc:3054] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.670266 15345 raft_consensus.cc:513] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.670399 15345 leader_election.cc:290] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:40.670603 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a3695e3b2849fdb1f6fa2e1fcfbc64" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:40.670675 15127 raft_consensus.cc:3054] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.670625 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a3695e3b2849fdb1f6fa2e1fcfbc64" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:40.670707 14862 raft_consensus.cc:3054] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.671202 15127 raft_consensus.cc:2463] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:40.671365 14532 leader_election.cc:304] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:40.671490 15345 raft_consensus.cc:2798] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.671521 14862 raft_consensus.cc:2463] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:40.671640 15345 raft_consensus.cc:695] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [term 1 LEADER]: Becoming Leader. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Running, Role: LEADER
I20250110 03:06:40.671725 15345 consensus_queue.cc:237] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:40.672452 14454 catalog_manager.cc:5526] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "97baa553877a44d2b1af2685ea7941a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:40.756162 15552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:40.756352 15552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:40.757649 15552 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20250110 03:06:40.789999 14994 raft_consensus.cc:1270] T c86f946ddcd94c40ac5f6f3db19e6bdc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:40.790433 15530 consensus_queue.cc:1035] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.790863 14862 raft_consensus.cc:1270] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.791016 15258 raft_consensus.cc:1270] T c86f946ddcd94c40ac5f6f3db19e6bdc P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:40.791185 14994 raft_consensus.cc:1270] T 3c7dc51d764044e8a6df257169f4d175 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.791227 15584 consensus_queue.cc:1035] T c86f946ddcd94c40ac5f6f3db19e6bdc P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.791325 15127 raft_consensus.cc:1270] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.791368 15361 consensus_queue.cc:1035] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.791520 14731 raft_consensus.cc:1270] T 3c7dc51d764044e8a6df257169f4d175 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:40.792618 15550 consensus_queue.cc:1035] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.793067 15361 consensus_queue.cc:1035] T 3c7dc51d764044e8a6df257169f4d175 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:40.795509 15550 consensus_queue.cc:1035] T d7a3695e3b2849fdb1f6fa2e1fcfbc64 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 27.5055 ms
time per row: 0.0137527 ms
I20250110 03:06:40.810387 14457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:40.812032 15106 tablet_service.cc:1467] Processing CreateTablet for tablet e9c3ac0322e748c6abb47dc1f5819b35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.812034 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 189de409fe9c4dbba152834e63d1a902 (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.812062 14843 tablet_service.cc:1467] Processing CreateTablet for tablet e9c3ac0322e748c6abb47dc1f5819b35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.812176 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c3ac0322e748c6abb47dc1f5819b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.812175 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c3ac0322e748c6abb47dc1f5819b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.812175 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189de409fe9c4dbba152834e63d1a902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.813108 15339 tablet_bootstrap.cc:492] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:40.813217 14578 tablet_service.cc:1467] Processing CreateTablet for tablet e9c3ac0322e748c6abb47dc1f5819b35 (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.813344 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9c3ac0322e748c6abb47dc1f5819b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.813489 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 189de409fe9c4dbba152834e63d1a902 (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.813534 15339 tablet_bootstrap.cc:654] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.813585 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189de409fe9c4dbba152834e63d1a902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.814172 15334 tablet_bootstrap.cc:492] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:40.814519 15330 tablet_bootstrap.cc:492] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:40.814538 15334 tablet_bootstrap.cc:654] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.814563 15338 tablet_bootstrap.cc:492] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:40.813120 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7ae54868ba418a91a671eac490864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.814990 15338 tablet_bootstrap.cc:654] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.814990 15330 tablet_bootstrap.cc:654] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.814989 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7ae54868ba418a91a671eac490864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.815256 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7ae54868ba418a91a671eac490864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.815344 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7ae54868ba418a91a671eac490864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.815347 15332 tablet_bootstrap.cc:492] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:40.815495 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 189de409fe9c4dbba152834e63d1a902 (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.815615 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189de409fe9c4dbba152834e63d1a902. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.816072 15332 tablet_bootstrap.cc:654] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.816609 15339 tablet_bootstrap.cc:492] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:40.816656 15339 ts_tablet_manager.cc:1397] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:06:40.816845 15338 tablet_bootstrap.cc:492] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:40.816821 15339 raft_consensus.cc:357] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.816891 15338 ts_tablet_manager.cc:1397] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:40.816898 15339 raft_consensus.cc:383] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.816920 15339 raft_consensus.cc:738] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.816922 15331 tablet_bootstrap.cc:492] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:40.816962 15330 tablet_bootstrap.cc:492] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:40.816965 15339 consensus_queue.cc:260] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.817003 15330 ts_tablet_manager.cc:1397] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:40.817067 15339 ts_tablet_manager.cc:1428] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:40.817034 15338 raft_consensus.cc:357] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.817121 15338 raft_consensus.cc:383] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.817137 15339 tablet_bootstrap.cc:492] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:40.817147 15338 raft_consensus.cc:738] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.817148 15330 raft_consensus.cc:357] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:40.817198 15338 consensus_queue.cc:260] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.817219 15330 raft_consensus.cc:383] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.817243 15330 raft_consensus.cc:738] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.817294 15338 ts_tablet_manager.cc:1428] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.817294 15330 consensus_queue.cc:260] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:40.817373 15331 tablet_bootstrap.cc:654] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.817404 15330 ts_tablet_manager.cc:1428] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:40.817555 15339 tablet_bootstrap.cc:654] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.818135 15339 tablet_bootstrap.cc:492] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:40.818176 15339 ts_tablet_manager.cc:1397] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.818346 15339 raft_consensus.cc:357] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.818398 15339 raft_consensus.cc:383] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.818421 15339 raft_consensus.cc:738] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.818470 15339 consensus_queue.cc:260] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.818610 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 9f7ae54868ba418a91a671eac490864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=346ccf79c7754d1e9f463716e2736363]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:40.818724 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7ae54868ba418a91a671eac490864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:40.818749 15339 ts_tablet_manager.cc:1428] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:40.819136 15331 tablet_bootstrap.cc:492] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:40.819166 15334 tablet_bootstrap.cc:492] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:40.819185 15331 ts_tablet_manager.cc:1397] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:40.819208 15334 ts_tablet_manager.cc:1397] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250110 03:06:40.819321 15331 raft_consensus.cc:357] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:40.819341 15334 raft_consensus.cc:357] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:40.819396 15331 raft_consensus.cc:383] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.819401 15334 raft_consensus.cc:383] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.819420 15331 raft_consensus.cc:738] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.819422 15334 raft_consensus.cc:738] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.819463 15331 consensus_queue.cc:260] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:40.819471 15334 consensus_queue.cc:260] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:40.819550 15331 ts_tablet_manager.cc:1428] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.819561 15334 ts_tablet_manager.cc:1428] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.819885 15522 raft_consensus.cc:491] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.819933 15338 tablet_bootstrap.cc:492] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:40.819940 15522 raft_consensus.cc:513] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.820075 15522 leader_election.cc:290] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:40.820338 15338 tablet_bootstrap.cc:654] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.820335 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ae54868ba418a91a671eac490864c" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:40.820422 15332 tablet_bootstrap.cc:492] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:40.820467 15332 ts_tablet_manager.cc:1397] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250110 03:06:40.820614 15332 raft_consensus.cc:357] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:40.820665 15332 raft_consensus.cc:383] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.820683 15332 raft_consensus.cc:738] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.820722 15332 consensus_queue.cc:260] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
W20250110 03:06:40.820760 14796 leader_election.cc:343] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589): Illegal state: must be running to vote when last-logged opid is not known
I20250110 03:06:40.820817 15332 ts_tablet_manager.cc:1428] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.820870 15332 tablet_bootstrap.cc:492] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:40.820896 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ae54868ba418a91a671eac490864c" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
W20250110 03:06:40.821099 14796 leader_election.cc:343] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181): Illegal state: must be running to vote when last-logged opid is not known
I20250110 03:06:40.821136 14796 leader_election.cc:304] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b; no voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1
I20250110 03:06:40.821224 15522 raft_consensus.cc:2743] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 03:06:40.821259 15332 tablet_bootstrap.cc:654] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:40.821923 15332 tablet_bootstrap.cc:492] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:40.821967 15332 ts_tablet_manager.cc:1397] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:40.822104 15332 raft_consensus.cc:357] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.822151 15332 raft_consensus.cc:383] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.822168 15332 raft_consensus.cc:738] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.822208 15332 consensus_queue.cc:260] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.822284 15332 ts_tablet_manager.cc:1428] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.822769 15338 tablet_bootstrap.cc:492] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:40.822826 15338 ts_tablet_manager.cc:1397] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:40.822948 15338 raft_consensus.cc:357] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.823004 15338 raft_consensus.cc:383] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:40.823024 15338 raft_consensus.cc:738] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:40.823083 15338 consensus_queue.cc:260] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.823194 15338 ts_tablet_manager.cc:1428] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:40.892463 15550 raft_consensus.cc:491] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:40.892589 15550 raft_consensus.cc:513] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.892762 15550 leader_election.cc:290] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:40.892954 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ae54868ba418a91a671eac490864c" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:40.892988 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ae54868ba418a91a671eac490864c" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:40.893057 14862 raft_consensus.cc:2463] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:40.893076 15127 raft_consensus.cc:2463] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:40.893239 14532 leader_election.cc:304] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:40.893333 15550 raft_consensus.cc:2798] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:40.893373 15550 raft_consensus.cc:491] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:40.893399 15550 raft_consensus.cc:3054] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.894039 15550 raft_consensus.cc:513] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.894162 15550 leader_election.cc:290] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:40.894321 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ae54868ba418a91a671eac490864c" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:40.894351 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f7ae54868ba418a91a671eac490864c" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:40.894392 14862 raft_consensus.cc:3054] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.894404 15127 raft_consensus.cc:3054] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:40.894994 15127 raft_consensus.cc:2463] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:40.894994 14862 raft_consensus.cc:2463] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:40.895164 14533 leader_election.cc:304] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:40.895305 15550 raft_consensus.cc:2798] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:40.895355 15550 raft_consensus.cc:695] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [term 1 LEADER]: Becoming Leader. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Running, Role: LEADER
I20250110 03:06:40.895403 15550 consensus_queue.cc:237] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:40.896065 14454 catalog_manager.cc:5526] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "97baa553877a44d2b1af2685ea7941a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:41.066707 15345 raft_consensus.cc:491] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.066825 15345 raft_consensus.cc:513] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.066977 15345 leader_election.cc:290] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:41.067193 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c3ac0322e748c6abb47dc1f5819b35" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:41.067296 15127 raft_consensus.cc:2463] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:41.067237 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c3ac0322e748c6abb47dc1f5819b35" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:41.067340 14862 raft_consensus.cc:2463] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:41.067441 14532 leader_election.cc:304] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:41.067544 15345 raft_consensus.cc:2798] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.067595 15345 raft_consensus.cc:491] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.067617 15345 raft_consensus.cc:3054] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.068248 15345 raft_consensus.cc:513] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.068384 15345 leader_election.cc:290] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:41.068563 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c3ac0322e748c6abb47dc1f5819b35" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:41.068563 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c3ac0322e748c6abb47dc1f5819b35" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:41.068639 14862 raft_consensus.cc:3054] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.068639 15127 raft_consensus.cc:3054] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.069327 14862 raft_consensus.cc:2463] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:41.069327 15127 raft_consensus.cc:2463] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:41.069526 14532 leader_election.cc:304] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:41.069634 15345 raft_consensus.cc:2798] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.069708 15345 raft_consensus.cc:695] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [term 1 LEADER]: Becoming Leader. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Running, Role: LEADER
I20250110 03:06:41.069766 15345 consensus_queue.cc:237] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.070384 14454 catalog_manager.cc:5526] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "97baa553877a44d2b1af2685ea7941a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:41.157812 15596 raft_consensus.cc:491] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.157922 15596 raft_consensus.cc:513] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:41.158089 15596 leader_election.cc:290] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:41.158308 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189de409fe9c4dbba152834e63d1a902" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:41.158339 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189de409fe9c4dbba152834e63d1a902" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:41.158396 14994 raft_consensus.cc:2463] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:41.158425 14731 raft_consensus.cc:2463] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:41.158557 15192 leader_election.cc:304] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.158696 15596 raft_consensus.cc:2798] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.158725 15596 raft_consensus.cc:491] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.158746 15596 raft_consensus.cc:3054] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.159361 15596 raft_consensus.cc:513] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:41.159485 15596 leader_election.cc:290] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:41.159657 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189de409fe9c4dbba152834e63d1a902" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:41.159724 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189de409fe9c4dbba152834e63d1a902" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:41.159775 14994 raft_consensus.cc:3054] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.159794 14731 raft_consensus.cc:3054] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.160574 14994 raft_consensus.cc:2463] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:41.160574 14731 raft_consensus.cc:2463] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:41.160753 15195 leader_election.cc:304] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.160856 15596 raft_consensus.cc:2798] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.160925 15596 raft_consensus.cc:695] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:41.160987 15596 consensus_queue.cc:237] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } }
I20250110 03:06:41.161602 14454 catalog_manager.cc:5526] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:41.250500 15609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:41.250710 15609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:41.251987 15609 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20250110 03:06:41.284937 14994 raft_consensus.cc:1270] T 189de409fe9c4dbba152834e63d1a902 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:41.285195 15596 consensus_queue.cc:1035] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.285270 14862 raft_consensus.cc:1270] T e9c3ac0322e748c6abb47dc1f5819b35 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:41.285470 14862 raft_consensus.cc:1270] T 9f7ae54868ba418a91a671eac490864c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.285475 15127 raft_consensus.cc:1270] T e9c3ac0322e748c6abb47dc1f5819b35 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:41.285562 15126 raft_consensus.cc:1270] T 9f7ae54868ba418a91a671eac490864c P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.285713 15550 consensus_queue.cc:1035] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.285753 15639 consensus_queue.cc:1035] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.285972 15345 consensus_queue.cc:1035] T e9c3ac0322e748c6abb47dc1f5819b35 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.286084 15345 consensus_queue.cc:1035] T 9f7ae54868ba418a91a671eac490864c P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.286412 14731 raft_consensus.cc:1270] T 189de409fe9c4dbba152834e63d1a902 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:41.286667 15596 consensus_queue.cc:1035] T 189de409fe9c4dbba152834e63d1a902 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 24.9879 ms
time per row: 0.0124939 ms
I20250110 03:06:41.302999 14454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:41.304594 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 5beceb1f2938452fbfaf4524cc204d5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.304594 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3924097f46449bbcae98eb394a9bbc (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.304687 14710 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3924097f46449bbcae98eb394a9bbc (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.304708 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 5beceb1f2938452fbfaf4524cc204d5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.304764 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5beceb1f2938452fbfaf4524cc204d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.304798 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a3924097f46449bbcae98eb394a9bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.304818 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5beceb1f2938452fbfaf4524cc204d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.304869 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a3924097f46449bbcae98eb394a9bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.305985 15330 tablet_bootstrap.cc:492] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:41.306061 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 7f62f52adc074581b1090285d43e3cf8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.306095 15331 tablet_bootstrap.cc:492] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:41.306144 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 5beceb1f2938452fbfaf4524cc204d5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.306192 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f62f52adc074581b1090285d43e3cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.306238 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5beceb1f2938452fbfaf4524cc204d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.306342 15330 tablet_bootstrap.cc:654] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.306480 15331 tablet_bootstrap.cc:654] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.307121 15331 tablet_bootstrap.cc:492] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:41.307166 15331 ts_tablet_manager.cc:1397] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:41.307354 15339 tablet_bootstrap.cc:492] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:41.307329 15331 raft_consensus.cc:357] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.307377 15334 tablet_bootstrap.cc:492] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:41.307400 15331 raft_consensus.cc:383] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.307422 15331 raft_consensus.cc:738] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.307482 15331 consensus_queue.cc:260] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.307585 15331 ts_tablet_manager.cc:1428] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:41.307716 15339 tablet_bootstrap.cc:654] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.307718 15334 tablet_bootstrap.cc:654] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.307879 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3924097f46449bbcae98eb394a9bbc (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.307859 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 7f62f52adc074581b1090285d43e3cf8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.307958 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a3924097f46449bbcae98eb394a9bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.307967 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f62f52adc074581b1090285d43e3cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.308169 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 7f62f52adc074581b1090285d43e3cf8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5b4b4edc50cd4899bd0dbd80249f107f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.308287 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f62f52adc074581b1090285d43e3cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.309041 15334 tablet_bootstrap.cc:492] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:41.309094 15334 ts_tablet_manager.cc:1397] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:06:41.309252 15332 tablet_bootstrap.cc:492] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:41.309238 15334 raft_consensus.cc:357] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.309293 15334 raft_consensus.cc:383] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.309285 15330 tablet_bootstrap.cc:492] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:41.309312 15334 raft_consensus.cc:738] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.309324 15330 ts_tablet_manager.cc:1397] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:41.309353 15334 consensus_queue.cc:260] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.309448 15334 ts_tablet_manager.cc:1428] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.309443 15330 raft_consensus.cc:357] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.309504 15330 raft_consensus.cc:383] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.309510 15334 tablet_bootstrap.cc:492] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:41.309530 15330 raft_consensus.cc:738] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.309583 15330 consensus_queue.cc:260] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.309616 15332 tablet_bootstrap.cc:654] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.309689 15330 ts_tablet_manager.cc:1428] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.309756 15330 tablet_bootstrap.cc:492] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:41.309935 15334 tablet_bootstrap.cc:654] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.310099 15331 tablet_bootstrap.cc:492] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:41.310163 15330 tablet_bootstrap.cc:654] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.310626 15338 tablet_bootstrap.cc:492] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:41.310721 15331 tablet_bootstrap.cc:654] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.311007 15339 tablet_bootstrap.cc:492] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:41.311013 15338 tablet_bootstrap.cc:654] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.311060 15339 ts_tablet_manager.cc:1397] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:06:41.311170 15339 raft_consensus.cc:357] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.311234 15339 raft_consensus.cc:383] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.311255 15339 raft_consensus.cc:738] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.311297 15339 consensus_queue.cc:260] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.311328 15332 tablet_bootstrap.cc:492] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:41.311375 15332 ts_tablet_manager.cc:1397] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:41.311403 15339 ts_tablet_manager.cc:1428] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.311537 15338 tablet_bootstrap.cc:492] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:41.311520 15332 raft_consensus.cc:357] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.311576 15338 ts_tablet_manager.cc:1397] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:41.311589 15332 raft_consensus.cc:383] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.311611 15332 raft_consensus.cc:738] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.311662 15332 consensus_queue.cc:260] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.311740 15334 tablet_bootstrap.cc:492] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:41.311774 15334 ts_tablet_manager.cc:1397] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250110 03:06:41.311771 15332 ts_tablet_manager.cc:1428] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.311744 15338 raft_consensus.cc:357] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.311803 15338 raft_consensus.cc:383] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.311826 15338 raft_consensus.cc:738] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.311877 15338 consensus_queue.cc:260] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.311929 15334 raft_consensus.cc:357] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.311976 15338 ts_tablet_manager.cc:1428] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.311990 15334 raft_consensus.cc:383] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.312012 15334 raft_consensus.cc:738] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.312054 15334 consensus_queue.cc:260] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.312142 15334 ts_tablet_manager.cc:1428] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.312356 15331 tablet_bootstrap.cc:492] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:41.312405 15331 ts_tablet_manager.cc:1397] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:41.312543 15331 raft_consensus.cc:357] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.312603 15331 raft_consensus.cc:383] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.312625 15331 raft_consensus.cc:738] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.312687 15331 consensus_queue.cc:260] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.312803 15331 ts_tablet_manager.cc:1428] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.313182 15330 tablet_bootstrap.cc:492] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:41.313236 15330 ts_tablet_manager.cc:1397] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:41.313396 15330 raft_consensus.cc:357] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.313481 15330 raft_consensus.cc:383] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.313504 15330 raft_consensus.cc:738] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.313542 15330 consensus_queue.cc:260] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.313658 15330 ts_tablet_manager.cc:1428] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.478376 15636 raft_consensus.cc:491] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.478497 15636 raft_consensus.cc:513] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.478741 15636 leader_election.cc:290] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:41.478961 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5beceb1f2938452fbfaf4524cc204d5d" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:41.478968 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5beceb1f2938452fbfaf4524cc204d5d" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:41.479063 14994 raft_consensus.cc:2463] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:41.479063 14731 raft_consensus.cc:2463] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:41.479207 15195 leader_election.cc:304] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.479310 15636 raft_consensus.cc:2798] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.479346 15636 raft_consensus.cc:491] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.479360 15636 raft_consensus.cc:3054] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.480020 15636 raft_consensus.cc:513] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.480136 15636 leader_election.cc:290] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:41.480309 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5beceb1f2938452fbfaf4524cc204d5d" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:41.480305 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5beceb1f2938452fbfaf4524cc204d5d" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:41.480392 14994 raft_consensus.cc:3054] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.480392 14731 raft_consensus.cc:3054] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.481146 14994 raft_consensus.cc:2463] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:41.481169 14731 raft_consensus.cc:2463] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:41.481344 15195 leader_election.cc:304] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.481447 15636 raft_consensus.cc:2798] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.481514 15636 raft_consensus.cc:695] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:41.481567 15636 consensus_queue.cc:237] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.482220 14451 catalog_manager.cc:5526] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:41.525409 15343 raft_consensus.cc:491] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.525519 15343 raft_consensus.cc:513] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.525815 15343 leader_election.cc:290] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:41.528815 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3924097f46449bbcae98eb394a9bbc" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:41.528846 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3924097f46449bbcae98eb394a9bbc" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:41.528930 14731 raft_consensus.cc:2463] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:41.528930 15258 raft_consensus.cc:2463] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:41.529120 14931 leader_election.cc:304] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.529245 15343 raft_consensus.cc:2798] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.529323 15343 raft_consensus.cc:491] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.529354 15343 raft_consensus.cc:3054] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.530076 15343 raft_consensus.cc:513] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.530215 15343 leader_election.cc:290] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Requested vote from peers c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:41.530400 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3924097f46449bbcae98eb394a9bbc" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:41.530434 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3924097f46449bbcae98eb394a9bbc" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:41.530479 14731 raft_consensus.cc:3054] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.530493 15258 raft_consensus.cc:3054] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.531038 14731 raft_consensus.cc:2463] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:41.531086 15258 raft_consensus.cc:2463] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:41.531205 14931 leader_election.cc:304] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:41.531289 15343 raft_consensus.cc:2798] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.531391 15343 raft_consensus.cc:695] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 LEADER]: Becoming Leader. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Running, Role: LEADER
I20250110 03:06:41.531471 15343 consensus_queue.cc:237] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:41.532176 14451 catalog_manager.cc:5526] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:41.656980 15642 raft_consensus.cc:491] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.657091 15642 raft_consensus.cc:513] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.657243 15642 leader_election.cc:290] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:41.657486 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f62f52adc074581b1090285d43e3cf8" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:41.657511 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f62f52adc074581b1090285d43e3cf8" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:41.657588 14598 raft_consensus.cc:2463] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:41.657586 15127 raft_consensus.cc:2463] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:41.657737 14796 leader_election.cc:304] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:41.657858 15642 raft_consensus.cc:2798] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.657912 15642 raft_consensus.cc:491] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.657933 15642 raft_consensus.cc:3054] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.658658 15642 raft_consensus.cc:513] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.658763 15642 leader_election.cc:290] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:41.658951 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f62f52adc074581b1090285d43e3cf8" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:41.658952 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f62f52adc074581b1090285d43e3cf8" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:41.659024 15127 raft_consensus.cc:3054] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.659022 14598 raft_consensus.cc:3054] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.659622 14598 raft_consensus.cc:2463] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:41.659624 15127 raft_consensus.cc:2463] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:41.659770 14796 leader_election.cc:304] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:41.659865 15642 raft_consensus.cc:2798] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.659921 15642 raft_consensus.cc:695] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Running, Role: LEADER
I20250110 03:06:41.659971 15642 consensus_queue.cc:237] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.660591 14451 catalog_manager.cc:5526] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131). New cstate: current_term: 1 leader_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:41.739979 15666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:41.740150 15666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:41.741333 15666 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20250110 03:06:41.774080 14731 raft_consensus.cc:1270] T 5beceb1f2938452fbfaf4524cc204d5d P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.774392 15127 raft_consensus.cc:1270] T 7f62f52adc074581b1090285d43e3cf8 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.774407 14598 raft_consensus.cc:1270] T 7f62f52adc074581b1090285d43e3cf8 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.774744 14994 raft_consensus.cc:1270] T 5beceb1f2938452fbfaf4524cc204d5d P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.774979 15522 consensus_queue.cc:1035] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.775038 15596 consensus_queue.cc:1035] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.775200 15596 consensus_queue.cc:1035] T 5beceb1f2938452fbfaf4524cc204d5d P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.776716 15522 consensus_queue.cc:1035] T 7f62f52adc074581b1090285d43e3cf8 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.780979 15258 raft_consensus.cc:1270] T 7a3924097f46449bbcae98eb394a9bbc P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.782538 14731 raft_consensus.cc:1270] T 7a3924097f46449bbcae98eb394a9bbc P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:41.782668 15663 consensus_queue.cc:1035] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:41.782778 15343 consensus_queue.cc:1035] T 7a3924097f46449bbcae98eb394a9bbc P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 27.0342 ms
time per row: 0.0135171 ms
I20250110 03:06:41.792845 14449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:41.794431 14842 tablet_service.cc:1467] Processing CreateTablet for tablet fa29c064a0eb4371ab4b1ba4aa99778b (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.794507 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 6f82b453356548b7839992e21f65dbf5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.794508 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 6f82b453356548b7839992e21f65dbf5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.794600 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa29c064a0eb4371ab4b1ba4aa99778b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.794621 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f82b453356548b7839992e21f65dbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.794656 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f82b453356548b7839992e21f65dbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.794721 14578 tablet_service.cc:1467] Processing CreateTablet for tablet fa29c064a0eb4371ab4b1ba4aa99778b (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.794818 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa29c064a0eb4371ab4b1ba4aa99778b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.795840 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3200d926b4744b6b01081984baae2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.795943 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d3200d926b4744b6b01081984baae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.795974 15332 tablet_bootstrap.cc:492] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:41.796062 15339 tablet_bootstrap.cc:492] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:41.796308 15332 tablet_bootstrap.cc:654] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.796386 15339 tablet_bootstrap.cc:654] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.796689 15106 tablet_service.cc:1467] Processing CreateTablet for tablet fa29c064a0eb4371ab4b1ba4aa99778b (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.796824 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa29c064a0eb4371ab4b1ba4aa99778b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.796833 15332 tablet_bootstrap.cc:492] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:41.796876 15332 ts_tablet_manager.cc:1397] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:41.797011 15332 raft_consensus.cc:357] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.797084 15332 raft_consensus.cc:383] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.797106 15332 raft_consensus.cc:738] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.797147 15332 consensus_queue.cc:260] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.797255 15332 ts_tablet_manager.cc:1428] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.797271 15330 tablet_bootstrap.cc:492] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:41.797308 15332 tablet_bootstrap.cc:492] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:41.797591 15332 tablet_bootstrap.cc:654] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.797668 15330 tablet_bootstrap.cc:654] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.797788 15338 tablet_bootstrap.cc:492] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:41.797824 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3200d926b4744b6b01081984baae2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.797902 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d3200d926b4744b6b01081984baae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.797968 15334 tablet_bootstrap.cc:492] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:41.798158 15338 tablet_bootstrap.cc:654] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.798362 15334 tablet_bootstrap.cc:654] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.798492 15330 tablet_bootstrap.cc:492] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:41.798542 15330 ts_tablet_manager.cc:1397] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:41.798748 15330 raft_consensus.cc:357] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.798810 15330 raft_consensus.cc:383] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.798848 15330 raft_consensus.cc:738] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.798892 15330 consensus_queue.cc:260] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.798938 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 6f82b453356548b7839992e21f65dbf5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.798995 15330 ts_tablet_manager.cc:1428] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.799062 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f82b453356548b7839992e21f65dbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.799628 15339 tablet_bootstrap.cc:492] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:41.799672 15339 ts_tablet_manager.cc:1397] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:06:41.799827 15339 raft_consensus.cc:357] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.799880 15339 raft_consensus.cc:383] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.799894 15339 raft_consensus.cc:738] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.799929 15339 consensus_queue.cc:260] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.800024 15339 ts_tablet_manager.cc:1428] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.800299 15331 tablet_bootstrap.cc:492] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:41.800357 15338 tablet_bootstrap.cc:492] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:41.800407 15338 ts_tablet_manager.cc:1397] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:41.800554 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 44d3200d926b4744b6b01081984baae2 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4f93bb7f28af4f3aa2ad2f14b034b5cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:41.800555 15338 raft_consensus.cc:357] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.800618 15338 raft_consensus.cc:383] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.800640 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d3200d926b4744b6b01081984baae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:41.800640 15338 raft_consensus.cc:738] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.800685 15338 consensus_queue.cc:260] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.800792 15338 ts_tablet_manager.cc:1428] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.800846 15338 tablet_bootstrap.cc:492] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:41.800854 15331 tablet_bootstrap.cc:654] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.801244 15338 tablet_bootstrap.cc:654] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.801482 15331 tablet_bootstrap.cc:492] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:41.801523 15331 ts_tablet_manager.cc:1397] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:41.801637 15331 raft_consensus.cc:357] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.801677 15331 raft_consensus.cc:383] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.801689 15331 raft_consensus.cc:738] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.801726 15331 consensus_queue.cc:260] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.801820 15331 ts_tablet_manager.cc:1428] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.801862 15339 tablet_bootstrap.cc:492] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:41.802134 15338 tablet_bootstrap.cc:492] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:41.802171 15338 ts_tablet_manager.cc:1397] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:41.802281 15339 tablet_bootstrap.cc:654] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:41.802294 15338 raft_consensus.cc:357] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.802345 15338 raft_consensus.cc:383] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.802366 15338 raft_consensus.cc:738] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.802431 15338 consensus_queue.cc:260] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.802522 15338 ts_tablet_manager.cc:1428] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.804015 15334 tablet_bootstrap.cc:492] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:41.804082 15334 ts_tablet_manager.cc:1397] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250110 03:06:41.804121 15339 tablet_bootstrap.cc:492] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:41.804174 15339 ts_tablet_manager.cc:1397] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:41.804220 15334 raft_consensus.cc:357] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.804304 15334 raft_consensus.cc:383] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.804327 15334 raft_consensus.cc:738] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.804334 15339 raft_consensus.cc:357] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.804396 15339 raft_consensus.cc:383] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.804378 15334 consensus_queue.cc:260] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.804416 15339 raft_consensus.cc:738] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.804461 15339 consensus_queue.cc:260] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.804497 15334 ts_tablet_manager.cc:1428] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.804543 15339 ts_tablet_manager.cc:1428] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:41.805220 15332 tablet_bootstrap.cc:492] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:41.805282 15332 ts_tablet_manager.cc:1397] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.000s
I20250110 03:06:41.805517 15332 raft_consensus.cc:357] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.805598 15332 raft_consensus.cc:383] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:41.805612 15332 raft_consensus.cc:738] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:41.805675 15332 consensus_queue.cc:260] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.805830 15332 ts_tablet_manager.cc:1428] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:41.864316 15698 raft_consensus.cc:491] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.864415 15698 raft_consensus.cc:513] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.864564 15698 leader_election.cc:290] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:41.864806 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f82b453356548b7839992e21f65dbf5" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:41.864817 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f82b453356548b7839992e21f65dbf5" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:41.864905 15258 raft_consensus.cc:2463] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:41.864905 14995 raft_consensus.cc:2463] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:41.865070 14667 leader_election.cc:304] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.865167 15698 raft_consensus.cc:2798] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.865209 15698 raft_consensus.cc:491] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.865231 15698 raft_consensus.cc:3054] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.865888 15698 raft_consensus.cc:513] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.866009 15698 leader_election.cc:290] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Requested vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:41.866261 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f82b453356548b7839992e21f65dbf5" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:41.866261 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f82b453356548b7839992e21f65dbf5" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:41.866338 14995 raft_consensus.cc:3054] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.866338 15258 raft_consensus.cc:3054] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.867069 15258 raft_consensus.cc:2463] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:41.867079 14995 raft_consensus.cc:2463] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:41.867269 14667 leader_election.cc:304] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:41.867388 15698 raft_consensus.cc:2798] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.867457 15698 raft_consensus.cc:695] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [term 1 LEADER]: Becoming Leader. State: Replica: c1502216bb0c469abd5491603f914b60, State: Running, Role: LEADER
I20250110 03:06:41.867503 15698 consensus_queue.cc:237] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:41.868182 14452 catalog_manager.cc:5526] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1502216bb0c469abd5491603f914b60 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "c1502216bb0c469abd5491603f914b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:41.879495 15344 raft_consensus.cc:491] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.879616 15344 raft_consensus.cc:513] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.879793 15344 leader_election.cc:290] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:41.879980 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29c064a0eb4371ab4b1ba4aa99778b" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:41.879988 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29c064a0eb4371ab4b1ba4aa99778b" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:41.880070 14598 raft_consensus.cc:2463] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:41.880081 14862 raft_consensus.cc:2463] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:41.880214 15060 leader_election.cc:304] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:41.880318 15344 raft_consensus.cc:2798] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.880371 15344 raft_consensus.cc:491] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.880393 15344 raft_consensus.cc:3054] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.881078 15344 raft_consensus.cc:513] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.881208 15344 leader_election.cc:290] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:41.881364 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29c064a0eb4371ab4b1ba4aa99778b" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:41.881366 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29c064a0eb4371ab4b1ba4aa99778b" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:41.881443 14862 raft_consensus.cc:3054] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.881443 14598 raft_consensus.cc:3054] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.882128 14598 raft_consensus.cc:2463] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:41.882128 14862 raft_consensus.cc:2463] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:41.882334 15060 leader_election.cc:304] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:41.882432 15344 raft_consensus.cc:2798] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.882586 15344 raft_consensus.cc:695] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 LEADER]: Becoming Leader. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Running, Role: LEADER
I20250110 03:06:41.882656 15344 consensus_queue.cc:237] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:41.883234 14452 catalog_manager.cc:5526] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:41.935127 15705 raft_consensus.cc:491] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:41.935217 15705 raft_consensus.cc:513] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.935370 15705 leader_election.cc:290] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:41.935628 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3200d926b4744b6b01081984baae2" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:41.935628 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3200d926b4744b6b01081984baae2" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:41.935714 14598 raft_consensus.cc:2463] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:41.935727 15127 raft_consensus.cc:2463] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:41.935865 14796 leader_election.cc:304] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:41.935979 15705 raft_consensus.cc:2798] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:41.936057 15705 raft_consensus.cc:491] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:41.936074 15705 raft_consensus.cc:3054] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.936734 15705 raft_consensus.cc:513] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.936867 15705 leader_election.cc:290] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:41.937021 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3200d926b4744b6b01081984baae2" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:41.937039 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d3200d926b4744b6b01081984baae2" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:41.937084 14598 raft_consensus.cc:3054] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.937093 15127 raft_consensus.cc:3054] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:41.937621 14598 raft_consensus.cc:2463] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:41.937716 15127 raft_consensus.cc:2463] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:41.937819 14796 leader_election.cc:304] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:41.937930 15705 raft_consensus.cc:2798] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:41.937990 15705 raft_consensus.cc:695] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Running, Role: LEADER
I20250110 03:06:41.938050 15705 consensus_queue.cc:237] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:41.938668 14452 catalog_manager.cc:5526] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131). New cstate: current_term: 1 leader_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:42.023402 15721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:42.023587 15721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:42.024868 15721 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20250110 03:06:42.057324 14598 raft_consensus.cc:1270] T fa29c064a0eb4371ab4b1ba4aa99778b P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.057718 15718 consensus_queue.cc:1035] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.057924 14862 raft_consensus.cc:1270] T fa29c064a0eb4371ab4b1ba4aa99778b P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.058477 15127 raft_consensus.cc:1270] T 44d3200d926b4744b6b01081984baae2 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.058779 15522 consensus_queue.cc:1035] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.058948 14599 raft_consensus.cc:1270] T 44d3200d926b4744b6b01081984baae2 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.059280 14995 raft_consensus.cc:1270] T 6f82b453356548b7839992e21f65dbf5 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.059412 15258 raft_consensus.cc:1270] T 6f82b453356548b7839992e21f65dbf5 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.059530 15584 consensus_queue.cc:1035] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.059581 15344 consensus_queue.cc:1035] T fa29c064a0eb4371ab4b1ba4aa99778b P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.059821 15584 consensus_queue.cc:1035] T 6f82b453356548b7839992e21f65dbf5 P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.061959 15705 consensus_queue.cc:1035] T 44d3200d926b4744b6b01081984baae2 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.065732 15531 mvcc.cc:204] Tried to move back new op lower bound from 7112615534828462080 to 7112615534338621440. Current Snapshot: MvccSnapshot[applied={T|T < 7112615534828462080}]
INSERT report
rows total: 2000
time total: 24.5519 ms
time per row: 0.0122759 ms
I20250110 03:06:42.076359 14450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:42.077947 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4106f93b444449b30af11ca704d8ce (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.077994 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4106f93b444449b30af11ca704d8ce (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.078121 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4106f93b444449b30af11ca704d8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.078104 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4106f93b444449b30af11ca704d8ce (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.078154 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4106f93b444449b30af11ca704d8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.078217 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4106f93b444449b30af11ca704d8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.079136 15334 tablet_bootstrap.cc:492] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:42.078009 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 063b02a7b5c44a6682cdcdcd6d0f9041 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.079293 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063b02a7b5c44a6682cdcdcd6d0f9041. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.079409 14710 tablet_service.cc:1467] Processing CreateTablet for tablet 063b02a7b5c44a6682cdcdcd6d0f9041 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.079408 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 063b02a7b5c44a6682cdcdcd6d0f9041 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.079494 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063b02a7b5c44a6682cdcdcd6d0f9041. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.079494 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063b02a7b5c44a6682cdcdcd6d0f9041. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.079589 15334 tablet_bootstrap.cc:654] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.080552 15330 tablet_bootstrap.cc:492] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:42.080619 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 5b25ee799c5a4062b980cb8eefde286c (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.080734 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b25ee799c5a4062b980cb8eefde286c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.081198 15330 tablet_bootstrap.cc:654] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.081533 15331 tablet_bootstrap.cc:492] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:42.081760 15330 tablet_bootstrap.cc:492] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:42.081804 15330 ts_tablet_manager.cc:1397] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:42.081926 15330 raft_consensus.cc:357] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.081941 15331 tablet_bootstrap.cc:654] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.081970 15330 raft_consensus.cc:383] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.081984 15330 raft_consensus.cc:738] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.082023 15330 consensus_queue.cc:260] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.082123 15330 ts_tablet_manager.cc:1428] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.082125 15332 tablet_bootstrap.cc:492] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:42.082170 15330 tablet_bootstrap.cc:492] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:42.082537 15332 tablet_bootstrap.cc:654] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.082624 15330 tablet_bootstrap.cc:654] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.083046 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 5b25ee799c5a4062b980cb8eefde286c (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.083180 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b25ee799c5a4062b980cb8eefde286c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.083196 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 5b25ee799c5a4062b980cb8eefde286c (DEFAULT_TABLE table=rebalance_test_table_7 [id=4bc78c15a37143e0b653403fcf1f1b2b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.083321 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b25ee799c5a4062b980cb8eefde286c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.083433 15334 tablet_bootstrap.cc:492] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:42.083477 15334 ts_tablet_manager.cc:1397] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250110 03:06:42.083621 15334 raft_consensus.cc:357] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.083675 15334 raft_consensus.cc:383] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.083693 15334 raft_consensus.cc:738] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.083765 15334 consensus_queue.cc:260] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.083868 15334 ts_tablet_manager.cc:1428] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.083925 15334 tablet_bootstrap.cc:492] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:42.084342 15334 tablet_bootstrap.cc:654] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.084365 15339 tablet_bootstrap.cc:492] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:42.084540 15338 tablet_bootstrap.cc:492] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:42.084765 15332 tablet_bootstrap.cc:492] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:42.084769 15339 tablet_bootstrap.cc:654] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.084810 15332 ts_tablet_manager.cc:1397] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:42.084992 15332 raft_consensus.cc:357] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.085036 15338 tablet_bootstrap.cc:654] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.085062 15332 raft_consensus.cc:383] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.085083 15332 raft_consensus.cc:738] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.085130 15332 consensus_queue.cc:260] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.085227 15332 ts_tablet_manager.cc:1428] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:42.085417 15334 tablet_bootstrap.cc:492] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:42.085458 15334 ts_tablet_manager.cc:1397] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:42.085605 15334 raft_consensus.cc:357] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.085675 15334 raft_consensus.cc:383] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.085700 15334 raft_consensus.cc:738] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.085742 15334 consensus_queue.cc:260] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.085839 15334 ts_tablet_manager.cc:1428] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.085858 15331 tablet_bootstrap.cc:492] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:42.085901 15331 ts_tablet_manager.cc:1397] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:06:42.085973 15338 tablet_bootstrap.cc:492] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:42.086016 15338 ts_tablet_manager.cc:1397] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:42.086086 15331 raft_consensus.cc:357] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.086166 15331 raft_consensus.cc:383] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.086196 15331 raft_consensus.cc:738] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.086189 15338 raft_consensus.cc:357] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.086247 15338 raft_consensus.cc:383] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.086267 15338 raft_consensus.cc:738] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.086247 15331 consensus_queue.cc:260] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.086318 15338 consensus_queue.cc:260] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.086349 15331 ts_tablet_manager.cc:1428] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.086401 15331 tablet_bootstrap.cc:492] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:42.086422 15338 ts_tablet_manager.cc:1428] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:42.086818 15330 tablet_bootstrap.cc:492] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:42.086853 15331 tablet_bootstrap.cc:654] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.086863 15330 ts_tablet_manager.cc:1397] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20250110 03:06:42.087009 15330 raft_consensus.cc:357] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.087072 15330 raft_consensus.cc:383] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.087091 15330 raft_consensus.cc:738] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.087136 15330 consensus_queue.cc:260] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.087237 15330 ts_tablet_manager.cc:1428] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.087759 15339 tablet_bootstrap.cc:492] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:42.087817 15339 ts_tablet_manager.cc:1397] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:42.087980 15339 raft_consensus.cc:357] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.088043 15339 raft_consensus.cc:383] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.088057 15339 raft_consensus.cc:738] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.088092 15339 consensus_queue.cc:260] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.088168 15339 ts_tablet_manager.cc:1428] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.088374 15331 tablet_bootstrap.cc:492] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:42.088421 15331 ts_tablet_manager.cc:1397] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:42.088697 15331 raft_consensus.cc:357] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.088775 15331 raft_consensus.cc:383] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.088807 15331 raft_consensus.cc:738] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.088858 15331 consensus_queue.cc:260] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.088958 15331 ts_tablet_manager.cc:1428] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:42.171329 15636 raft_consensus.cc:491] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.171427 15636 raft_consensus.cc:513] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.171566 15636 leader_election.cc:290] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.171780 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063b02a7b5c44a6682cdcdcd6d0f9041" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:42.171862 14731 raft_consensus.cc:2463] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:42.171803 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063b02a7b5c44a6682cdcdcd6d0f9041" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:42.171931 14995 raft_consensus.cc:2463] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:42.172013 15195 leader_election.cc:304] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.172116 15636 raft_consensus.cc:2798] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.172152 15636 raft_consensus.cc:491] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.172166 15636 raft_consensus.cc:3054] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.172873 15636 raft_consensus.cc:513] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.172996 15636 leader_election.cc:290] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.173153 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063b02a7b5c44a6682cdcdcd6d0f9041" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:42.173147 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "063b02a7b5c44a6682cdcdcd6d0f9041" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:42.173224 14731 raft_consensus.cc:3054] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.173224 14995 raft_consensus.cc:3054] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.173938 14995 raft_consensus.cc:2463] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:42.173938 14731 raft_consensus.cc:2463] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:42.174118 15195 leader_election.cc:304] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.174234 15636 raft_consensus.cc:2798] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.174291 15636 raft_consensus.cc:695] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:42.174347 15636 consensus_queue.cc:237] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.174978 14450 catalog_manager.cc:5526] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:42.202859 15705 raft_consensus.cc:491] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.202958 15705 raft_consensus.cc:513] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.203133 15705 leader_election.cc:290] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:42.203321 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b25ee799c5a4062b980cb8eefde286c" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:42.203331 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b25ee799c5a4062b980cb8eefde286c" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:42.203418 14598 raft_consensus.cc:2463] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:42.203419 15127 raft_consensus.cc:2463] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:42.203576 14796 leader_election.cc:304] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:42.203675 15705 raft_consensus.cc:2798] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.203718 15705 raft_consensus.cc:491] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.203740 15705 raft_consensus.cc:3054] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.204356 15705 raft_consensus.cc:513] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.204483 15705 leader_election.cc:290] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:42.204622 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b25ee799c5a4062b980cb8eefde286c" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:42.204636 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b25ee799c5a4062b980cb8eefde286c" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:42.204680 15127 raft_consensus.cc:3054] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.204695 14598 raft_consensus.cc:3054] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.205235 15127 raft_consensus.cc:2463] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:42.205291 14598 raft_consensus.cc:2463] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:42.205382 14796 leader_election.cc:304] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:42.205466 15705 raft_consensus.cc:2798] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.205519 15705 raft_consensus.cc:695] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Running, Role: LEADER
I20250110 03:06:42.205575 15705 consensus_queue.cc:237] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.206337 14450 catalog_manager.cc:5526] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131). New cstate: current_term: 1 leader_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:42.251293 15663 raft_consensus.cc:491] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.251403 15663 raft_consensus.cc:513] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.251557 15663 leader_election.cc:290] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.251852 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4106f93b444449b30af11ca704d8ce" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:42.251828 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4106f93b444449b30af11ca704d8ce" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:42.251950 14731 raft_consensus.cc:2463] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:42.251950 15258 raft_consensus.cc:2463] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:42.252135 14931 leader_election.cc:304] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.252228 15663 raft_consensus.cc:2798] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.252274 15663 raft_consensus.cc:491] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.252296 15663 raft_consensus.cc:3054] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.253008 15663 raft_consensus.cc:513] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.253126 15663 leader_election.cc:290] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Requested vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.253306 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4106f93b444449b30af11ca704d8ce" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:42.253389 15258 raft_consensus.cc:3054] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.253355 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4106f93b444449b30af11ca704d8ce" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:42.253451 14731 raft_consensus.cc:3054] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.254237 14731 raft_consensus.cc:2463] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:42.254235 15258 raft_consensus.cc:2463] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:42.254434 14931 leader_election.cc:304] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:42.254513 15663 raft_consensus.cc:2798] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.254599 15663 raft_consensus.cc:695] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 LEADER]: Becoming Leader. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Running, Role: LEADER
I20250110 03:06:42.254655 15663 consensus_queue.cc:237] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.255264 14450 catalog_manager.cc:5526] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:42.341168 15771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:42.341322 15771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:42.342551 15771 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20250110 03:06:42.375568 14995 raft_consensus.cc:1270] T 063b02a7b5c44a6682cdcdcd6d0f9041 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.376199 15546 consensus_queue.cc:1035] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.376255 14731 raft_consensus.cc:1270] T 063b02a7b5c44a6682cdcdcd6d0f9041 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.376498 15258 raft_consensus.cc:1270] T 3a4106f93b444449b30af11ca704d8ce P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.376564 15546 consensus_queue.cc:1035] T 063b02a7b5c44a6682cdcdcd6d0f9041 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.376747 14730 raft_consensus.cc:1270] T 3a4106f93b444449b30af11ca704d8ce P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.376912 14598 raft_consensus.cc:1270] T 5b25ee799c5a4062b980cb8eefde286c P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.377085 15127 raft_consensus.cc:1270] T 5b25ee799c5a4062b980cb8eefde286c P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.377758 15343 consensus_queue.cc:1035] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.378121 15522 consensus_queue.cc:1035] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.378250 15522 consensus_queue.cc:1035] T 5b25ee799c5a4062b980cb8eefde286c P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.380841 15343 consensus_queue.cc:1035] T 3a4106f93b444449b30af11ca704d8ce P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.383791 15397 mvcc.cc:204] Tried to move back new op lower bound from 7112615536128053248 to 7112615535306412032. Current Snapshot: MvccSnapshot[applied={T|T < 7112615536128053248}]
INSERT report
rows total: 2000
time total: 28.6296 ms
time per row: 0.0143148 ms
I20250110 03:06:42.391000 15808 mvcc.cc:204] Tried to move back new op lower bound from 7112615536138432512 to 7112615535635476480. Current Snapshot: MvccSnapshot[applied={T|T < 7112615536138432512 or (T in {7112615536138432512})}]
I20250110 03:06:42.396586 14457 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:42.398133 14579 tablet_service.cc:1467] Processing CreateTablet for tablet a980455a026745e79dd30be0c49daf0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.398142 15107 tablet_service.cc:1467] Processing CreateTablet for tablet a980455a026745e79dd30be0c49daf0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.398138 15239 tablet_service.cc:1467] Processing CreateTablet for tablet c99d57111378432f8f5bb80d38615db2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.398242 14842 tablet_service.cc:1467] Processing CreateTablet for tablet a980455a026745e79dd30be0c49daf0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.398272 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a980455a026745e79dd30be0c49daf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.398272 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a980455a026745e79dd30be0c49daf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.398272 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99d57111378432f8f5bb80d38615db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.398342 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a980455a026745e79dd30be0c49daf0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.399585 15338 tablet_bootstrap.cc:492] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:42.399580 14975 tablet_service.cc:1467] Processing CreateTablet for tablet c99d57111378432f8f5bb80d38615db2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.399698 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99d57111378432f8f5bb80d38615db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.399724 14842 tablet_service.cc:1467] Processing CreateTablet for tablet c36a81205a014aeb836fe41beef43cdd (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.399801 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36a81205a014aeb836fe41beef43cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.399802 15332 tablet_bootstrap.cc:492] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:42.400051 15338 tablet_bootstrap.cc:654] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.400144 15332 tablet_bootstrap.cc:654] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.400579 14579 tablet_service.cc:1467] Processing CreateTablet for tablet c36a81205a014aeb836fe41beef43cdd (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.400681 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36a81205a014aeb836fe41beef43cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.400907 15331 tablet_bootstrap.cc:492] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:42.400941 15334 tablet_bootstrap.cc:492] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:42.401320 15334 tablet_bootstrap.cc:654] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.401466 15331 tablet_bootstrap.cc:654] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.401613 15339 tablet_bootstrap.cc:492] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:42.401751 15338 tablet_bootstrap.cc:492] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:42.401794 15338 ts_tablet_manager.cc:1397] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:42.401819 15334 tablet_bootstrap.cc:492] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:42.401856 15334 ts_tablet_manager.cc:1397] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:42.402016 15339 tablet_bootstrap.cc:654] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.402011 15334 raft_consensus.cc:357] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.402096 15334 raft_consensus.cc:383] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.402072 15338 raft_consensus.cc:357] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.402128 15334 raft_consensus.cc:738] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.402145 15338 raft_consensus.cc:383] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.402166 15338 raft_consensus.cc:738] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.402160 15334 consensus_queue.cc:260] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.402199 15331 tablet_bootstrap.cc:492] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:42.402216 15338 consensus_queue.cc:260] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.402241 15331 ts_tablet_manager.cc:1397] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:42.402267 15334 ts_tablet_manager.cc:1428] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.402297 15338 ts_tablet_manager.cc:1428] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:42.402377 15331 raft_consensus.cc:357] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.402431 15331 raft_consensus.cc:383] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.402417 14710 tablet_service.cc:1467] Processing CreateTablet for tablet c99d57111378432f8f5bb80d38615db2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.402451 15331 raft_consensus.cc:738] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.402490 15331 consensus_queue.cc:260] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.402518 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99d57111378432f8f5bb80d38615db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.402611 15331 ts_tablet_manager.cc:1428] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:42.403728 15330 tablet_bootstrap.cc:492] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:42.404023 15107 tablet_service.cc:1467] Processing CreateTablet for tablet c36a81205a014aeb836fe41beef43cdd (DEFAULT_TABLE table=rebalance_test_table_8 [id=346fe633e08049b286bf9a79d325898a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.404143 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c36a81205a014aeb836fe41beef43cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.404141 15330 tablet_bootstrap.cc:654] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.404547 15339 tablet_bootstrap.cc:492] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:42.404587 15339 ts_tablet_manager.cc:1397] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:42.404724 15330 tablet_bootstrap.cc:492] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:42.404757 15330 ts_tablet_manager.cc:1397] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:42.404732 15339 raft_consensus.cc:357] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.404793 15339 raft_consensus.cc:383] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.404831 15339 raft_consensus.cc:738] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.404863 15330 raft_consensus.cc:357] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.404882 15339 consensus_queue.cc:260] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.404917 15330 raft_consensus.cc:383] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.404939 15330 raft_consensus.cc:738] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.404971 15339 ts_tablet_manager.cc:1428] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.404980 15330 consensus_queue.cc:260] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.405027 15339 tablet_bootstrap.cc:492] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:42.405081 15330 ts_tablet_manager.cc:1428] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.405349 15338 tablet_bootstrap.cc:492] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:42.405432 15339 tablet_bootstrap.cc:654] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.405757 15338 tablet_bootstrap.cc:654] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.406152 15339 tablet_bootstrap.cc:492] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:42.406198 15339 ts_tablet_manager.cc:1397] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:42.406333 15339 raft_consensus.cc:357] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.406404 15339 raft_consensus.cc:383] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.406432 15339 raft_consensus.cc:738] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.406474 15339 consensus_queue.cc:260] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.406597 15339 ts_tablet_manager.cc:1428] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.406715 15338 tablet_bootstrap.cc:492] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:42.406759 15338 ts_tablet_manager.cc:1397] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:42.406818 15332 tablet_bootstrap.cc:492] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:42.406876 15332 ts_tablet_manager.cc:1397] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20250110 03:06:42.406914 15338 raft_consensus.cc:357] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.406999 15338 raft_consensus.cc:383] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.407019 15338 raft_consensus.cc:738] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.407016 15332 raft_consensus.cc:357] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.407083 15332 raft_consensus.cc:383] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.407066 15338 consensus_queue.cc:260] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.407104 15332 raft_consensus.cc:738] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.407159 15338 ts_tablet_manager.cc:1428] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.407160 15332 consensus_queue.cc:260] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.407269 15332 ts_tablet_manager.cc:1428] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.407332 15332 tablet_bootstrap.cc:492] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:42.407757 15332 tablet_bootstrap.cc:654] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.408550 15332 tablet_bootstrap.cc:492] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:42.408601 15332 ts_tablet_manager.cc:1397] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:42.408731 15332 raft_consensus.cc:357] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.408777 15332 raft_consensus.cc:383] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.408790 15332 raft_consensus.cc:738] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.408818 15332 consensus_queue.cc:260] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.408890 15332 ts_tablet_manager.cc:1428] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.424249 15344 raft_consensus.cc:491] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.424338 15344 raft_consensus.cc:513] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.424476 15344 leader_election.cc:290] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:42.424669 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a81205a014aeb836fe41beef43cdd" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:42.424775 14862 raft_consensus.cc:2463] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:42.424782 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a81205a014aeb836fe41beef43cdd" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:42.424896 14598 raft_consensus.cc:2463] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:42.424932 15061 leader_election.cc:304] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:42.425026 15344 raft_consensus.cc:2798] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.425067 15344 raft_consensus.cc:491] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.425091 15344 raft_consensus.cc:3054] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.425659 15344 raft_consensus.cc:513] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.425786 15344 leader_election.cc:290] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Requested vote from peers 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:42.425927 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a81205a014aeb836fe41beef43cdd" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:42.425944 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c36a81205a014aeb836fe41beef43cdd" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:42.425985 14862 raft_consensus.cc:3054] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.426009 14598 raft_consensus.cc:3054] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.426621 14862 raft_consensus.cc:2463] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:42.426649 14598 raft_consensus.cc:2463] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:42.426781 15061 leader_election.cc:304] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:42.426857 15344 raft_consensus.cc:2798] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.426909 15344 raft_consensus.cc:695] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 LEADER]: Becoming Leader. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Running, Role: LEADER
I20250110 03:06:42.426966 15344 consensus_queue.cc:237] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:42.427549 14448 catalog_manager.cc:5526] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:42.444980 15522 raft_consensus.cc:491] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.445086 15522 raft_consensus.cc:513] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.445262 15522 leader_election.cc:290] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:42.445446 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980455a026745e79dd30be0c49daf0f" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:42.445484 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980455a026745e79dd30be0c49daf0f" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:42.445538 15127 raft_consensus.cc:2463] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:42.445554 14598 raft_consensus.cc:2463] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:42.445896 14796 leader_election.cc:304] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:42.445981 15522 raft_consensus.cc:2798] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.446039 15522 raft_consensus.cc:491] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.446069 15522 raft_consensus.cc:3054] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.446794 15522 raft_consensus.cc:513] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.446913 15522 leader_election.cc:290] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589)
I20250110 03:06:42.447080 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980455a026745e79dd30be0c49daf0f" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:42.447085 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a980455a026745e79dd30be0c49daf0f" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:42.447163 15127 raft_consensus.cc:3054] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.447166 14598 raft_consensus.cc:3054] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.447674 14598 raft_consensus.cc:2463] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:42.447857 14796 leader_election.cc:304] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:42.447939 15522 raft_consensus.cc:2798] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.448033 15522 raft_consensus.cc:695] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Running, Role: LEADER
I20250110 03:06:42.448081 15522 consensus_queue.cc:237] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } }
I20250110 03:06:42.448698 14448 catalog_manager.cc:5526] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131). New cstate: current_term: 1 leader_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:42.467260 15127 raft_consensus.cc:2463] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:42.481372 15343 raft_consensus.cc:491] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.481477 15343 raft_consensus.cc:513] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.481635 15343 leader_election.cc:290] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.481868 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99d57111378432f8f5bb80d38615db2" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:42.481894 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99d57111378432f8f5bb80d38615db2" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:42.481963 15258 raft_consensus.cc:2463] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:42.481971 14731 raft_consensus.cc:2463] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:42.482141 14931 leader_election.cc:304] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:42.482264 15343 raft_consensus.cc:2798] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.482316 15343 raft_consensus.cc:491] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.482331 15343 raft_consensus.cc:3054] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.483096 15343 raft_consensus.cc:513] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.483209 15343 leader_election.cc:290] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Requested vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.483381 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99d57111378432f8f5bb80d38615db2" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:42.483399 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99d57111378432f8f5bb80d38615db2" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:42.483453 14731 raft_consensus.cc:3054] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.483448 15258 raft_consensus.cc:3054] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.483938 14731 raft_consensus.cc:2463] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:42.484001 15258 raft_consensus.cc:2463] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:42.484095 14931 leader_election.cc:304] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:42.484187 15343 raft_consensus.cc:2798] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.484236 15343 raft_consensus.cc:695] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 LEADER]: Becoming Leader. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Running, Role: LEADER
I20250110 03:06:42.484302 15343 consensus_queue.cc:237] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.484923 14448 catalog_manager.cc:5526] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:42.574393 15825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:42.574605 15825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:42.575917 15825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20250110 03:06:42.608929 14731 raft_consensus.cc:1270] T c99d57111378432f8f5bb80d38615db2 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.609122 15258 raft_consensus.cc:1270] T c99d57111378432f8f5bb80d38615db2 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.609472 14598 raft_consensus.cc:1270] T a980455a026745e79dd30be0c49daf0f P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.609539 15663 consensus_queue.cc:1035] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.609632 15127 raft_consensus.cc:1270] T a980455a026745e79dd30be0c49daf0f P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.609681 15343 consensus_queue.cc:1035] T c99d57111378432f8f5bb80d38615db2 P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.609928 15705 consensus_queue.cc:1035] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.610060 15705 consensus_queue.cc:1035] T a980455a026745e79dd30be0c49daf0f P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.610671 14862 raft_consensus.cc:1270] T c36a81205a014aeb836fe41beef43cdd P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.610893 15821 consensus_queue.cc:1035] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.611332 14598 raft_consensus.cc:1270] T c36a81205a014aeb836fe41beef43cdd P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:42.611568 15821 consensus_queue.cc:1035] T c36a81205a014aeb836fe41beef43cdd P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:42.613669 15703 mvcc.cc:204] Tried to move back new op lower bound from 7112615537084133376 to 7112615536575967232. Current Snapshot: MvccSnapshot[applied={T|T < 7112615537084133376}]
INSERT report
rows total: 2000
time total: 29.6063 ms
time per row: 0.0148032 ms
I20250110 03:06:42.633809 14448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:42.633924 15533 mvcc.cc:204] Tried to move back new op lower bound from 7112615537084850176 to 7112615536427732992. Current Snapshot: MvccSnapshot[applied={T|T < 7112615537084850176}]
I20250110 03:06:42.635515 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 39b64221fb1746ed877fe589490c2f40 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.635587 14710 tablet_service.cc:1467] Processing CreateTablet for tablet 39b64221fb1746ed877fe589490c2f40 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.635715 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39b64221fb1746ed877fe589490c2f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.635701 14974 tablet_service.cc:1467] Processing CreateTablet for tablet f7bc887900f345438f29ce2c17cac992 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.635761 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39b64221fb1746ed877fe589490c2f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.635550 15238 tablet_service.cc:1467] Processing CreateTablet for tablet f7bc887900f345438f29ce2c17cac992 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.635758 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 739cecd3905a48eca63b1dbff431afb5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.635828 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bc887900f345438f29ce2c17cac992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.635864 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bc887900f345438f29ce2c17cac992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.635901 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 739cecd3905a48eca63b1dbff431afb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.636922 15331 tablet_bootstrap.cc:492] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:42.636950 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 39b64221fb1746ed877fe589490c2f40 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.637029 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39b64221fb1746ed877fe589490c2f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.637115 15339 tablet_bootstrap.cc:492] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:42.637123 15330 tablet_bootstrap.cc:492] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:42.637416 15331 tablet_bootstrap.cc:654] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.637518 15339 tablet_bootstrap.cc:654] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.637518 15330 tablet_bootstrap.cc:654] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.638067 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 739cecd3905a48eca63b1dbff431afb5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.638185 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 739cecd3905a48eca63b1dbff431afb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.638163 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 739cecd3905a48eca63b1dbff431afb5 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.638268 14711 tablet_service.cc:1467] Processing CreateTablet for tablet f7bc887900f345438f29ce2c17cac992 (DEFAULT_TABLE table=rebalance_test_table_9 [id=869e112f52f044729fef4c8daa7cd0a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:42.638288 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 739cecd3905a48eca63b1dbff431afb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.638352 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bc887900f345438f29ce2c17cac992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:42.639314 15338 tablet_bootstrap.cc:492] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:42.639461 15332 tablet_bootstrap.cc:492] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:42.639638 15339 tablet_bootstrap.cc:492] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:42.639688 15339 ts_tablet_manager.cc:1397] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:42.639804 15338 tablet_bootstrap.cc:654] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.639815 15331 tablet_bootstrap.cc:492] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:42.639861 15331 ts_tablet_manager.cc:1397] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:06:42.639851 15339 raft_consensus.cc:357] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.639900 15332 tablet_bootstrap.cc:654] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.639930 15339 raft_consensus.cc:383] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.639950 15339 raft_consensus.cc:738] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.639997 15339 consensus_queue.cc:260] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.640012 15331 raft_consensus.cc:357] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.640089 15331 raft_consensus.cc:383] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.640096 15339 ts_tablet_manager.cc:1428] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:42.640120 15331 raft_consensus.cc:738] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.640161 15331 consensus_queue.cc:260] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.640269 15331 ts_tablet_manager.cc:1428] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:42.640327 15331 tablet_bootstrap.cc:492] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:42.640484 15338 tablet_bootstrap.cc:492] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:42.640517 15338 ts_tablet_manager.cc:1397] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:42.640616 15332 tablet_bootstrap.cc:492] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:42.640619 15338 raft_consensus.cc:357] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.640657 15338 raft_consensus.cc:383] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.640671 15338 raft_consensus.cc:738] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.640666 15332 ts_tablet_manager.cc:1397] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:42.640709 15338 consensus_queue.cc:260] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.640810 15338 ts_tablet_manager.cc:1428] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.640801 15332 raft_consensus.cc:357] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.640858 15331 tablet_bootstrap.cc:654] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.640883 15332 raft_consensus.cc:383] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.640905 15332 raft_consensus.cc:738] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.640962 15332 consensus_queue.cc:260] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.641072 15332 ts_tablet_manager.cc:1428] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.659749 15334 tablet_bootstrap.cc:492] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:42.660207 15334 tablet_bootstrap.cc:654] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.660840 15334 tablet_bootstrap.cc:492] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:42.660889 15334 ts_tablet_manager.cc:1397] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:42.661070 15334 raft_consensus.cc:357] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.661149 15334 raft_consensus.cc:383] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.661162 15334 raft_consensus.cc:738] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.661204 15334 consensus_queue.cc:260] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.661314 15334 ts_tablet_manager.cc:1428] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.661372 15334 tablet_bootstrap.cc:492] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:42.661742 15334 tablet_bootstrap.cc:654] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.662322 15334 tablet_bootstrap.cc:492] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:42.662361 15334 ts_tablet_manager.cc:1397] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:42.662604 15334 raft_consensus.cc:357] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.662667 15334 raft_consensus.cc:383] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.662681 15334 raft_consensus.cc:738] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.662734 15334 consensus_queue.cc:260] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.662839 15334 ts_tablet_manager.cc:1428] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.666713 15330 tablet_bootstrap.cc:492] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:42.666723 15331 tablet_bootstrap.cc:492] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:42.666775 15330 ts_tablet_manager.cc:1397] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.001s
I20250110 03:06:42.666770 15331 ts_tablet_manager.cc:1397] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.001s
I20250110 03:06:42.666936 15330 raft_consensus.cc:357] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.666949 15331 raft_consensus.cc:357] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.666994 15330 raft_consensus.cc:383] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.667014 15331 raft_consensus.cc:383] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.667019 15330 raft_consensus.cc:738] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.667034 15331 raft_consensus.cc:738] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.667097 15330 consensus_queue.cc:260] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.667114 15331 consensus_queue.cc:260] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.667225 15331 ts_tablet_manager.cc:1428] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.667237 15330 ts_tablet_manager.cc:1428] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.667313 15330 tablet_bootstrap.cc:492] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:42.668001 15330 tablet_bootstrap.cc:654] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:42.668540 15330 tablet_bootstrap.cc:492] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:42.668583 15330 ts_tablet_manager.cc:1397] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:42.668694 15330 raft_consensus.cc:357] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.668742 15330 raft_consensus.cc:383] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:42.668766 15330 raft_consensus.cc:738] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:42.668821 15330 consensus_queue.cc:260] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.668910 15330 ts_tablet_manager.cc:1428] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:42.703330 15663 raft_consensus.cc:491] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.703442 15663 raft_consensus.cc:513] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.703603 15663 leader_election.cc:290] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:42.703878 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bc887900f345438f29ce2c17cac992" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:42.703835 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bc887900f345438f29ce2c17cac992" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:42.703981 15258 raft_consensus.cc:2463] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:42.703987 14731 raft_consensus.cc:2463] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:42.704128 14931 leader_election.cc:304] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.704224 15663 raft_consensus.cc:2798] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.704268 15663 raft_consensus.cc:491] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.704289 15663 raft_consensus.cc:3054] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.704963 15663 raft_consensus.cc:513] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.705071 15663 leader_election.cc:290] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Requested vote from peers c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113), e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333)
I20250110 03:06:42.705266 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bc887900f345438f29ce2c17cac992" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:42.705250 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bc887900f345438f29ce2c17cac992" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:42.705330 15258 raft_consensus.cc:3054] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.705340 14731 raft_consensus.cc:3054] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.705840 15258 raft_consensus.cc:2463] T f7bc887900f345438f29ce2c17cac992 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:42.705924 14731 raft_consensus.cc:2463] T f7bc887900f345438f29ce2c17cac992 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:42.706004 14931 leader_election.cc:304] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.706113 15663 raft_consensus.cc:2798] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.706164 15663 raft_consensus.cc:695] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 LEADER]: Becoming Leader. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Running, Role: LEADER
I20250110 03:06:42.706216 15663 consensus_queue.cc:237] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } }
I20250110 03:06:42.706888 14450 catalog_manager.cc:5526] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:42.735718 15705 raft_consensus.cc:491] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.735824 15705 raft_consensus.cc:513] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.735963 15705 leader_election.cc:290] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:42.736192 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739cecd3905a48eca63b1dbff431afb5" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:42.736184 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739cecd3905a48eca63b1dbff431afb5" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:42.736296 14599 raft_consensus.cc:2463] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:42.736296 15127 raft_consensus.cc:2463] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 0.
I20250110 03:06:42.736454 14796 leader_election.cc:304] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:42.736562 15705 raft_consensus.cc:2798] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.736615 15705 raft_consensus.cc:491] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.736650 15705 raft_consensus.cc:3054] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.737293 15705 raft_consensus.cc:513] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.737427 15705 leader_election.cc:290] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181)
I20250110 03:06:42.737586 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739cecd3905a48eca63b1dbff431afb5" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:42.737599 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739cecd3905a48eca63b1dbff431afb5" candidate_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:42.737663 15127 raft_consensus.cc:3054] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.737663 14599 raft_consensus.cc:3054] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.738366 15127 raft_consensus.cc:2463] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:42.738366 14599 raft_consensus.cc:2463] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1352ba3b2b4c47a4aa6c3e77f1418b in term 1.
I20250110 03:06:42.738538 14796 leader_election.cc:304] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:42.738662 15705 raft_consensus.cc:2798] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.738723 15705 raft_consensus.cc:695] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Running, Role: LEADER
I20250110 03:06:42.738777 15705 consensus_queue.cc:237] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:42.739413 14450 catalog_manager.cc:5526] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131). New cstate: current_term: 1 leader_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:42.945724 15636 raft_consensus.cc:491] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:42.945827 15636 raft_consensus.cc:513] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.946017 15636 leader_election.cc:290] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.946244 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b64221fb1746ed877fe589490c2f40" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:42.946245 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b64221fb1746ed877fe589490c2f40" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:42.946331 14995 raft_consensus.cc:2463] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:42.946343 14731 raft_consensus.cc:2463] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:42.946509 15192 leader_election.cc:304] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.946620 15636 raft_consensus.cc:2798] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:42.946672 15636 raft_consensus.cc:491] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:42.946699 15636 raft_consensus.cc:3054] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.947350 15636 raft_consensus.cc:513] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.947481 15636 leader_election.cc:290] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:42.947626 14731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b64221fb1746ed877fe589490c2f40" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:42.947628 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b64221fb1746ed877fe589490c2f40" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:42.947685 14995 raft_consensus.cc:3054] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.947687 14731 raft_consensus.cc:3054] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:42.948218 14731 raft_consensus.cc:2463] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:42.948220 14995 raft_consensus.cc:2463] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:42.948400 15192 leader_election.cc:304] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:42.948499 15636 raft_consensus.cc:2798] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:42.948617 15636 raft_consensus.cc:695] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:42.948670 15636 consensus_queue.cc:237] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:42.949223 14450 catalog_manager.cc:5526] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:43.071322 15877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:43.071501 15877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:43.072772 15877 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20250110 03:06:43.085294 15663 consensus_queue.cc:1035] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.086959 15663 consensus_queue.cc:1035] T f7bc887900f345438f29ce2c17cac992 P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.104144 15127 raft_consensus.cc:1270] T 739cecd3905a48eca63b1dbff431afb5 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.105608 14599 raft_consensus.cc:1270] T 739cecd3905a48eca63b1dbff431afb5 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer 4c1352ba3b2b4c47a4aa6c3e77f1418b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.106709 14995 raft_consensus.cc:1270] T 39b64221fb1746ed877fe589490c2f40 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.107583 14730 raft_consensus.cc:1270] T 39b64221fb1746ed877fe589490c2f40 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.108175 15705 consensus_queue.cc:1035] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.108333 15705 consensus_queue.cc:1035] T 739cecd3905a48eca63b1dbff431afb5 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.108451 15636 consensus_queue.cc:1035] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.108593 15875 consensus_queue.cc:1035] T 39b64221fb1746ed877fe589490c2f40 P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.112324 15401 mvcc.cc:204] Tried to move back new op lower bound from 7112615539124854784 to 7112615537618399232. Current Snapshot: MvccSnapshot[applied={T|T < 7112615539111800832}]
INSERT report
rows total: 2000
time total: 24.975 ms
time per row: 0.0124875 ms
I20250110 03:06:43.124421 14449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:43.126044 15107 tablet_service.cc:1467] Processing CreateTablet for tablet 4d846f8b51fb4098858434052d13a696 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.126050 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 4d846f8b51fb4098858434052d13a696 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.126113 15106 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcf43c1a9ec48d7ba9d50954ea07769 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.126168 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 689dc9359eed458193b4e089fb19fdab (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.126236 15106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcf43c1a9ec48d7ba9d50954ea07769. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.126237 14579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d846f8b51fb4098858434052d13a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.126286 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689dc9359eed458193b4e089fb19fdab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.126483 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d846f8b51fb4098858434052d13a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.126050 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcf43c1a9ec48d7ba9d50954ea07769 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.127579 15332 tablet_bootstrap.cc:492] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:43.127614 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcf43c1a9ec48d7ba9d50954ea07769. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.127624 15331 tablet_bootstrap.cc:492] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:43.127727 15338 tablet_bootstrap.cc:492] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:43.128072 15338 tablet_bootstrap.cc:654] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.128073 15331 tablet_bootstrap.cc:654] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.128079 15332 tablet_bootstrap.cc:654] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.128706 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 689dc9359eed458193b4e089fb19fdab (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.128748 14842 tablet_service.cc:1467] Processing CreateTablet for tablet 4d846f8b51fb4098858434052d13a696 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.128827 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689dc9359eed458193b4e089fb19fdab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.128881 14842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d846f8b51fb4098858434052d13a696. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.128899 15331 tablet_bootstrap.cc:492] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:43.128937 15331 ts_tablet_manager.cc:1397] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:43.128927 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 689dc9359eed458193b4e089fb19fdab (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.129031 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689dc9359eed458193b4e089fb19fdab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.129093 15331 raft_consensus.cc:357] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.129184 15331 raft_consensus.cc:383] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.129204 15331 raft_consensus.cc:738] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.129254 15331 consensus_queue.cc:260] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.129340 15331 ts_tablet_manager.cc:1428] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.129714 15338 tablet_bootstrap.cc:492] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:43.129753 15338 ts_tablet_manager.cc:1397] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:43.129873 15338 raft_consensus.cc:357] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.129925 15338 raft_consensus.cc:383] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.129943 15338 raft_consensus.cc:738] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.129980 15338 consensus_queue.cc:260] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.130089 15338 ts_tablet_manager.cc:1428] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.130116 14843 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcf43c1a9ec48d7ba9d50954ea07769 (DEFAULT_TABLE table=rebalance_test_table_10 [id=e4c4cc0988b247629f2644c9206a145f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.130153 15338 tablet_bootstrap.cc:492] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:43.130213 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcf43c1a9ec48d7ba9d50954ea07769. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.130468 15338 tablet_bootstrap.cc:654] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.130506 15330 tablet_bootstrap.cc:492] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:43.130868 15339 tablet_bootstrap.cc:492] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:43.131008 15330 tablet_bootstrap.cc:654] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.131289 15339 tablet_bootstrap.cc:654] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.131395 15334 tablet_bootstrap.cc:492] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:43.131480 15332 tablet_bootstrap.cc:492] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:43.131522 15332 ts_tablet_manager.cc:1397] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:06:43.131683 15332 raft_consensus.cc:357] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.131754 15332 raft_consensus.cc:383] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.131779 15332 raft_consensus.cc:738] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.131816 15334 tablet_bootstrap.cc:654] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.131824 15339 tablet_bootstrap.cc:492] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:43.131825 15332 consensus_queue.cc:260] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.131871 15339 ts_tablet_manager.cc:1397] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:43.131953 15332 ts_tablet_manager.cc:1428] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.132016 15332 tablet_bootstrap.cc:492] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:43.132022 15339 raft_consensus.cc:357] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.132093 15339 raft_consensus.cc:383] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.132112 15339 raft_consensus.cc:738] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.132144 15339 consensus_queue.cc:260] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.132238 15339 ts_tablet_manager.cc:1428] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.132292 15339 tablet_bootstrap.cc:492] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:43.132422 15332 tablet_bootstrap.cc:654] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.132685 15339 tablet_bootstrap.cc:654] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.133550 15332 tablet_bootstrap.cc:492] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:43.133603 15332 ts_tablet_manager.cc:1397] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:43.133608 15334 tablet_bootstrap.cc:492] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:43.133658 15334 ts_tablet_manager.cc:1397] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:43.133673 15339 tablet_bootstrap.cc:492] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:43.133710 15339 ts_tablet_manager.cc:1397] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:43.133769 15332 raft_consensus.cc:357] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.133785 15334 raft_consensus.cc:357] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.133841 15334 raft_consensus.cc:383] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.133839 15332 raft_consensus.cc:383] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.133855 15334 raft_consensus.cc:738] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.133862 15332 raft_consensus.cc:738] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.133858 15339 raft_consensus.cc:357] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.133913 15339 raft_consensus.cc:383] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.133900 15334 consensus_queue.cc:260] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.133939 15339 raft_consensus.cc:738] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.133916 15332 consensus_queue.cc:260] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.133989 15339 consensus_queue.cc:260] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.134017 15332 ts_tablet_manager.cc:1428] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.134021 15334 ts_tablet_manager.cc:1428] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:43.134088 15339 ts_tablet_manager.cc:1428] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.134297 15330 tablet_bootstrap.cc:492] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:43.134349 15330 ts_tablet_manager.cc:1397] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250110 03:06:43.134510 15330 raft_consensus.cc:357] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.134610 15330 raft_consensus.cc:383] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.134634 15330 raft_consensus.cc:738] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.134683 15330 consensus_queue.cc:260] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.134717 15338 tablet_bootstrap.cc:492] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:43.134756 15338 ts_tablet_manager.cc:1397] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250110 03:06:43.134785 15330 ts_tablet_manager.cc:1428] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.134896 15338 raft_consensus.cc:357] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.134963 15338 raft_consensus.cc:383] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.134990 15338 raft_consensus.cc:738] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.135033 15338 consensus_queue.cc:260] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.135151 15338 ts_tablet_manager.cc:1428] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:43.300633 15638 raft_consensus.cc:491] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:43.300721 15638 raft_consensus.cc:513] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.300880 15638 leader_election.cc:290] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:43.301046 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcf43c1a9ec48d7ba9d50954ea07769" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:43.301131 15127 raft_consensus.cc:2463] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:43.301223 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcf43c1a9ec48d7ba9d50954ea07769" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:43.301270 14532 leader_election.cc:304] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:43.301304 14862 raft_consensus.cc:2463] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:43.301363 15638 raft_consensus.cc:2798] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:43.301406 15638 raft_consensus.cc:491] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:43.301429 15638 raft_consensus.cc:3054] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.302073 15638 raft_consensus.cc:513] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.302197 15638 leader_election.cc:290] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:43.302354 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcf43c1a9ec48d7ba9d50954ea07769" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:43.302423 15127 raft_consensus.cc:3054] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.302523 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcf43c1a9ec48d7ba9d50954ea07769" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:43.302624 14862 raft_consensus.cc:3054] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.303004 15127 raft_consensus.cc:2463] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:43.303167 14862 raft_consensus.cc:2463] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:43.303181 14532 leader_election.cc:304] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:43.303277 15638 raft_consensus.cc:2798] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:43.303408 15638 raft_consensus.cc:695] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [term 1 LEADER]: Becoming Leader. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Running, Role: LEADER
I20250110 03:06:43.303462 15638 consensus_queue.cc:237] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.304059 14454 catalog_manager.cc:5526] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "97baa553877a44d2b1af2685ea7941a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:43.336881 15875 raft_consensus.cc:491] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:43.336987 15875 raft_consensus.cc:513] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.337158 15875 leader_election.cc:290] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:43.337402 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689dc9359eed458193b4e089fb19fdab" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:43.337405 14730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689dc9359eed458193b4e089fb19fdab" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:43.337489 14995 raft_consensus.cc:2463] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:43.337522 14730 raft_consensus.cc:2463] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 0.
I20250110 03:06:43.337631 15192 leader_election.cc:304] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:43.337734 15875 raft_consensus.cc:2798] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:43.337774 15875 raft_consensus.cc:491] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:43.337790 15875 raft_consensus.cc:3054] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.338398 15875 raft_consensus.cc:513] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.338508 15875 leader_election.cc:290] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Requested vote from peers 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:43.338697 14730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689dc9359eed458193b4e089fb19fdab" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:43.338697 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "689dc9359eed458193b4e089fb19fdab" candidate_uuid: "e7ff2606284a4b339d7d9b7e74256850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:43.338771 14730 raft_consensus.cc:3054] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.338771 14995 raft_consensus.cc:3054] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.339473 14730 raft_consensus.cc:2463] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:43.339473 14995 raft_consensus.cc:2463] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7ff2606284a4b339d7d9b7e74256850 in term 1.
I20250110 03:06:43.339676 15192 leader_election.cc:304] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:43.339782 15875 raft_consensus.cc:2798] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:43.339844 15875 raft_consensus.cc:695] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Running, Role: LEADER
I20250110 03:06:43.339901 15875 consensus_queue.cc:237] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.340541 14454 catalog_manager.cc:5526] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "e7ff2606284a4b339d7d9b7e74256850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:43.394071 15638 raft_consensus.cc:491] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:43.394184 15638 raft_consensus.cc:513] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.394317 15638 leader_election.cc:290] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:43.394527 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d846f8b51fb4098858434052d13a696" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" is_pre_election: true
I20250110 03:06:43.394539 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d846f8b51fb4098858434052d13a696" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:43.394639 15127 raft_consensus.cc:2463] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:43.394639 14862 raft_consensus.cc:2463] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 0.
I20250110 03:06:43.394798 14533 leader_election.cc:304] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, 97baa553877a44d2b1af2685ea7941a8; no voters:
I20250110 03:06:43.394893 15638 raft_consensus.cc:2798] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:43.394969 15638 raft_consensus.cc:491] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:43.394987 15638 raft_consensus.cc:3054] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.395648 15638 raft_consensus.cc:513] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.395763 15638 leader_election.cc:290] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Requested vote from peers c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133:42181), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:43.395902 15127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d846f8b51fb4098858434052d13a696" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63cf2b9cefd498f8106f39dc06e88d1"
I20250110 03:06:43.395923 14862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d846f8b51fb4098858434052d13a696" candidate_uuid: "97baa553877a44d2b1af2685ea7941a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:43.395979 15127 raft_consensus.cc:3054] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.395989 14862 raft_consensus.cc:3054] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.396700 15127 raft_consensus.cc:2463] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:43.396723 14862 raft_consensus.cc:2463] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97baa553877a44d2b1af2685ea7941a8 in term 1.
I20250110 03:06:43.396898 14532 leader_election.cc:304] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97baa553877a44d2b1af2685ea7941a8, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:43.397028 15638 raft_consensus.cc:2798] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:43.397094 15638 raft_consensus.cc:695] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [term 1 LEADER]: Becoming Leader. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Running, Role: LEADER
I20250110 03:06:43.397146 15638 consensus_queue.cc:237] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } }
I20250110 03:06:43.397790 14454 catalog_manager.cc:5526] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "97baa553877a44d2b1af2685ea7941a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:43.490100 15930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:43.490271 15930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:43.491539 15930 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20250110 03:06:43.526304 14730 raft_consensus.cc:1270] T 689dc9359eed458193b4e089fb19fdab P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:43.526304 14995 raft_consensus.cc:1270] T 689dc9359eed458193b4e089fb19fdab P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer e7ff2606284a4b339d7d9b7e74256850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:43.526587 15127 raft_consensus.cc:1270] T 4d846f8b51fb4098858434052d13a696 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:43.526597 15960 consensus_queue.cc:1035] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.526651 14863 raft_consensus.cc:1270] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.526715 15126 raft_consensus.cc:1270] T 6fcf43c1a9ec48d7ba9d50954ea07769 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.526738 14862 raft_consensus.cc:1270] T 4d846f8b51fb4098858434052d13a696 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer 97baa553877a44d2b1af2685ea7941a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:43.526883 15636 consensus_queue.cc:1035] T 689dc9359eed458193b4e089fb19fdab P e7ff2606284a4b339d7d9b7e74256850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.527117 15926 consensus_queue.cc:1035] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.527297 15926 consensus_queue.cc:1035] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.527441 15926 consensus_queue.cc:1035] T 6fcf43c1a9ec48d7ba9d50954ea07769 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.527551 15926 consensus_queue.cc:1035] T 4d846f8b51fb4098858434052d13a696 P 97baa553877a44d2b1af2685ea7941a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 27.7727 ms
time per row: 0.0138863 ms
I20250110 03:06:43.544062 14449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:43.545681 15238 tablet_service.cc:1467] Processing CreateTablet for tablet 134365cffdb647ce9a1447a3f1b81418 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.545745 14710 tablet_service.cc:1467] Processing CreateTablet for tablet 1a17e23216384550ac12b7b725f82b5a (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.545784 14975 tablet_service.cc:1467] Processing CreateTablet for tablet 134365cffdb647ce9a1447a3f1b81418 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.545830 15238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134365cffdb647ce9a1447a3f1b81418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.545893 14975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134365cffdb647ce9a1447a3f1b81418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.545893 14710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a17e23216384550ac12b7b725f82b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.546865 15334 tablet_bootstrap.cc:492] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:43.546888 15239 tablet_service.cc:1467] Processing CreateTablet for tablet 1a17e23216384550ac12b7b725f82b5a (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.546963 15239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a17e23216384550ac12b7b725f82b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.547148 15331 tablet_bootstrap.cc:492] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:43.547184 15107 tablet_service.cc:1467] Processing CreateTablet for tablet cbb2879d5da745deb40b4e9da557ff40 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.547293 15107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbb2879d5da745deb40b4e9da557ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.547307 15334 tablet_bootstrap.cc:654] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.547523 15331 tablet_bootstrap.cc:654] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.547991 14711 tablet_service.cc:1467] Processing CreateTablet for tablet 134365cffdb647ce9a1447a3f1b81418 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.547993 14843 tablet_service.cc:1467] Processing CreateTablet for tablet cbb2879d5da745deb40b4e9da557ff40 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.548087 14843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbb2879d5da745deb40b4e9da557ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.548095 14711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 134365cffdb647ce9a1447a3f1b81418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.548157 15331 tablet_bootstrap.cc:492] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:43.548197 15331 ts_tablet_manager.cc:1397] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:43.548328 15331 raft_consensus.cc:357] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.548386 15331 raft_consensus.cc:383] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.548409 15331 raft_consensus.cc:738] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.548456 15331 consensus_queue.cc:260] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.548558 15331 ts_tablet_manager.cc:1428] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.548645 15330 tablet_bootstrap.cc:492] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:43.548979 15330 tablet_bootstrap.cc:654] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.549233 15338 tablet_bootstrap.cc:492] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1: Bootstrap starting.
I20250110 03:06:43.549219 14578 tablet_service.cc:1467] Processing CreateTablet for tablet cbb2879d5da745deb40b4e9da557ff40 (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.549263 15339 tablet_bootstrap.cc:492] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Bootstrap starting.
I20250110 03:06:43.549326 14578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbb2879d5da745deb40b4e9da557ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.549602 15338 tablet_bootstrap.cc:654] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.549640 15339 tablet_bootstrap.cc:654] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.550148 15330 tablet_bootstrap.cc:492] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:43.550179 15330 ts_tablet_manager.cc:1397] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:43.550280 15330 raft_consensus.cc:357] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.550320 15330 raft_consensus.cc:383] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.550333 15330 raft_consensus.cc:738] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.550369 15330 consensus_queue.cc:260] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.550390 15339 tablet_bootstrap.cc:492] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: No bootstrap required, opened a new log
I20250110 03:06:43.550426 15339 ts_tablet_manager.cc:1397] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:43.550467 15330 ts_tablet_manager.cc:1428] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.550514 15330 tablet_bootstrap.cc:492] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60: Bootstrap starting.
I20250110 03:06:43.550714 15332 tablet_bootstrap.cc:492] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8: Bootstrap starting.
I20250110 03:06:43.550717 15339 raft_consensus.cc:357] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.550756 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 1a17e23216384550ac12b7b725f82b5a (DEFAULT_TABLE table=rebalance_test_table_11 [id=0ca2547927b040a293b4db4768abae48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:43.550804 15339 raft_consensus.cc:383] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.550850 15339 raft_consensus.cc:738] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1352ba3b2b4c47a4aa6c3e77f1418b, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.550881 14974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a17e23216384550ac12b7b725f82b5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:43.550894 15339 consensus_queue.cc:260] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.550951 15330 tablet_bootstrap.cc:654] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.550981 15339 ts_tablet_manager.cc:1428] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:43.551121 15332 tablet_bootstrap.cc:654] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.551489 15338 tablet_bootstrap.cc:492] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1: No bootstrap required, opened a new log
I20250110 03:06:43.551533 15338 ts_tablet_manager.cc:1397] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:43.551666 15338 raft_consensus.cc:357] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.551721 15338 raft_consensus.cc:383] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.551735 15338 raft_consensus.cc:738] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.551772 15338 consensus_queue.cc:260] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.551882 15338 ts_tablet_manager.cc:1428] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.552181 15330 tablet_bootstrap.cc:492] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60: No bootstrap required, opened a new log
I20250110 03:06:43.552208 15331 tablet_bootstrap.cc:492] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05: Bootstrap starting.
I20250110 03:06:43.552218 15330 ts_tablet_manager.cc:1397] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:43.552362 15330 raft_consensus.cc:357] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.552415 15330 raft_consensus.cc:383] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.552433 15330 raft_consensus.cc:738] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1502216bb0c469abd5491603f914b60, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.552466 15330 consensus_queue.cc:260] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.552541 15330 ts_tablet_manager.cc:1428] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.552553 15331 tablet_bootstrap.cc:654] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.553025 15332 tablet_bootstrap.cc:492] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8: No bootstrap required, opened a new log
I20250110 03:06:43.553066 15332 ts_tablet_manager.cc:1397] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:43.553205 15332 raft_consensus.cc:357] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.553264 15332 raft_consensus.cc:383] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.553284 15332 raft_consensus.cc:738] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97baa553877a44d2b1af2685ea7941a8, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.553323 15332 consensus_queue.cc:260] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.553438 15332 ts_tablet_manager.cc:1428] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.553642 15331 tablet_bootstrap.cc:492] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05: No bootstrap required, opened a new log
I20250110 03:06:43.553709 15331 ts_tablet_manager.cc:1397] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:06:43.553889 15331 raft_consensus.cc:357] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.553966 15331 raft_consensus.cc:383] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.553988 15331 raft_consensus.cc:738] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.554031 15331 consensus_queue.cc:260] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.554145 15331 ts_tablet_manager.cc:1428] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.554608 15334 tablet_bootstrap.cc:492] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:43.554657 15334 ts_tablet_manager.cc:1397] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20250110 03:06:43.554790 15334 raft_consensus.cc:357] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.554843 15334 raft_consensus.cc:383] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.554865 15334 raft_consensus.cc:738] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.554912 15334 consensus_queue.cc:260] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.554999 15334 ts_tablet_manager.cc:1428] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.555079 15334 tablet_bootstrap.cc:492] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850: Bootstrap starting.
I20250110 03:06:43.555434 15334 tablet_bootstrap.cc:654] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:43.556125 15334 tablet_bootstrap.cc:492] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850: No bootstrap required, opened a new log
I20250110 03:06:43.556186 15334 ts_tablet_manager.cc:1397] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:43.556311 15334 raft_consensus.cc:357] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.556363 15334 raft_consensus.cc:383] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:43.556387 15334 raft_consensus.cc:738] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff2606284a4b339d7d9b7e74256850, State: Initialized, Role: FOLLOWER
I20250110 03:06:43.556440 15334 consensus_queue.cc:260] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.556527 15334 ts_tablet_manager.cc:1428] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:43.592216 15910 raft_consensus.cc:491] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:43.592312 15910 raft_consensus.cc:513] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.592465 15910 leader_election.cc:290] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:43.592713 14730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a17e23216384550ac12b7b725f82b5a" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60" is_pre_election: true
I20250110 03:06:43.592687 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a17e23216384550ac12b7b725f82b5a" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:43.592823 14730 raft_consensus.cc:2463] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:43.592839 15258 raft_consensus.cc:2463] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 0.
I20250110 03:06:43.592983 14931 leader_election.cc:304] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:43.593099 15910 raft_consensus.cc:2798] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:43.593147 15910 raft_consensus.cc:491] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:43.593171 15910 raft_consensus.cc:3054] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.593852 15910 raft_consensus.cc:513] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.593977 15910 leader_election.cc:290] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Requested vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), c1502216bb0c469abd5491603f914b60 (127.14.19.130:45113)
I20250110 03:06:43.594159 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a17e23216384550ac12b7b725f82b5a" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:43.594170 14730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a17e23216384550ac12b7b725f82b5a" candidate_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1502216bb0c469abd5491603f914b60"
I20250110 03:06:43.594233 14730 raft_consensus.cc:3054] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.594231 15258 raft_consensus.cc:3054] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.594825 15258 raft_consensus.cc:2463] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:43.594839 14730 raft_consensus.cc:2463] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830d0fd8356b411e9ad8df94d9c9ab05 in term 1.
I20250110 03:06:43.594996 14931 leader_election.cc:304] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:43.595099 15910 raft_consensus.cc:2798] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:43.595155 15910 raft_consensus.cc:695] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 LEADER]: Becoming Leader. State: Replica: 830d0fd8356b411e9ad8df94d9c9ab05, State: Running, Role: LEADER
I20250110 03:06:43.595204 15910 consensus_queue.cc:237] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.595839 14449 catalog_manager.cc:5526] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:43.602279 15968 raft_consensus.cc:491] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:43.602375 15968 raft_consensus.cc:513] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.602562 15968 leader_election.cc:290] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:43.602721 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb2879d5da745deb40b4e9da557ff40" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" is_pre_election: true
I20250110 03:06:43.602756 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb2879d5da745deb40b4e9da557ff40" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8" is_pre_election: true
I20250110 03:06:43.602810 14863 raft_consensus.cc:2463] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:43.602823 14599 raft_consensus.cc:2463] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 0.
I20250110 03:06:43.602944 15061 leader_election.cc:304] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:43.603042 15968 raft_consensus.cc:2798] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:43.603097 15968 raft_consensus.cc:491] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:43.603123 15968 raft_consensus.cc:3054] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.603771 15968 raft_consensus.cc:513] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.603911 15968 leader_election.cc:290] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Requested vote from peers 97baa553877a44d2b1af2685ea7941a8 (127.14.19.129:44589), 4c1352ba3b2b4c47a4aa6c3e77f1418b (127.14.19.131:33887)
I20250110 03:06:43.604029 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb2879d5da745deb40b4e9da557ff40" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97baa553877a44d2b1af2685ea7941a8"
I20250110 03:06:43.604032 14863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbb2879d5da745deb40b4e9da557ff40" candidate_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b"
I20250110 03:06:43.604094 14863 raft_consensus.cc:3054] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.604094 14599 raft_consensus.cc:3054] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.604652 14863 raft_consensus.cc:2463] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:43.604713 14599 raft_consensus.cc:2463] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c63cf2b9cefd498f8106f39dc06e88d1 in term 1.
I20250110 03:06:43.604830 15061 leader_election.cc:304] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1352ba3b2b4c47a4aa6c3e77f1418b, c63cf2b9cefd498f8106f39dc06e88d1; no voters:
I20250110 03:06:43.604907 15968 raft_consensus.cc:2798] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:43.604959 15968 raft_consensus.cc:695] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [term 1 LEADER]: Becoming Leader. State: Replica: c63cf2b9cefd498f8106f39dc06e88d1, State: Running, Role: LEADER
I20250110 03:06:43.605006 15968 consensus_queue.cc:237] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } }
I20250110 03:06:43.605610 14449 catalog_manager.cc:5526] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c63cf2b9cefd498f8106f39dc06e88d1 (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c63cf2b9cefd498f8106f39dc06e88d1" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42181 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:43.614465 15698 raft_consensus.cc:491] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:43.614563 15698 raft_consensus.cc:513] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.614761 15698 leader_election.cc:290] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:43.614981 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134365cffdb647ce9a1447a3f1b81418" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" is_pre_election: true
I20250110 03:06:43.615072 14995 raft_consensus.cc:2463] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:43.615065 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134365cffdb647ce9a1447a3f1b81418" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850" is_pre_election: true
I20250110 03:06:43.615185 15258 raft_consensus.cc:2463] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 0.
I20250110 03:06:43.615250 14664 leader_election.cc:304] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830d0fd8356b411e9ad8df94d9c9ab05, c1502216bb0c469abd5491603f914b60; no voters:
I20250110 03:06:43.615334 15698 raft_consensus.cc:2798] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:43.615391 15698 raft_consensus.cc:491] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:43.615450 15698 raft_consensus.cc:3054] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.616144 15698 raft_consensus.cc:513] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.616268 15698 leader_election.cc:290] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Requested vote from peers e7ff2606284a4b339d7d9b7e74256850 (127.14.19.134:37333), 830d0fd8356b411e9ad8df94d9c9ab05 (127.14.19.132:46299)
I20250110 03:06:43.616412 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134365cffdb647ce9a1447a3f1b81418" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830d0fd8356b411e9ad8df94d9c9ab05"
I20250110 03:06:43.616412 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134365cffdb647ce9a1447a3f1b81418" candidate_uuid: "c1502216bb0c469abd5491603f914b60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7ff2606284a4b339d7d9b7e74256850"
I20250110 03:06:43.616480 15258 raft_consensus.cc:3054] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.616487 14995 raft_consensus.cc:3054] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:43.617210 15258 raft_consensus.cc:2463] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:43.617233 14995 raft_consensus.cc:2463] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1502216bb0c469abd5491603f914b60 in term 1.
I20250110 03:06:43.617367 14667 leader_election.cc:304] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1502216bb0c469abd5491603f914b60, e7ff2606284a4b339d7d9b7e74256850; no voters:
I20250110 03:06:43.617455 15698 raft_consensus.cc:2798] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:43.617575 15698 raft_consensus.cc:695] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [term 1 LEADER]: Becoming Leader. State: Replica: c1502216bb0c469abd5491603f914b60, State: Running, Role: LEADER
I20250110 03:06:43.617616 15698 consensus_queue.cc:237] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } }
I20250110 03:06:43.618253 14449 catalog_manager.cc:5526] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1502216bb0c469abd5491603f914b60 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "c1502216bb0c469abd5491603f914b60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 } health_report { overall_health: HEALTHY } } }
W20250110 03:06:43.703596 15989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:43.703769 15989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:43.704986 15989 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20250110 03:06:43.735342 14863 raft_consensus.cc:1270] T cbb2879d5da745deb40b4e9da557ff40 P 4c1352ba3b2b4c47a4aa6c3e77f1418b [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.735355 14599 raft_consensus.cc:1270] T cbb2879d5da745deb40b4e9da557ff40 P 97baa553877a44d2b1af2685ea7941a8 [term 1 FOLLOWER]: Refusing update from remote peer c63cf2b9cefd498f8106f39dc06e88d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.735786 15968 consensus_queue.cc:1035] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1352ba3b2b4c47a4aa6c3e77f1418b" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.735957 15968 consensus_queue.cc:1035] T cbb2879d5da745deb40b4e9da557ff40 P c63cf2b9cefd498f8106f39dc06e88d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97baa553877a44d2b1af2685ea7941a8" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.736889 14995 raft_consensus.cc:1270] T 134365cffdb647ce9a1447a3f1b81418 P 830d0fd8356b411e9ad8df94d9c9ab05 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.737059 15258 raft_consensus.cc:1270] T 134365cffdb647ce9a1447a3f1b81418 P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer c1502216bb0c469abd5491603f914b60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.737345 15987 consensus_queue.cc:1035] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830d0fd8356b411e9ad8df94d9c9ab05" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.737859 15987 consensus_queue.cc:1035] T 134365cffdb647ce9a1447a3f1b81418 P c1502216bb0c469abd5491603f914b60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.738150 14730 raft_consensus.cc:1270] T 1a17e23216384550ac12b7b725f82b5a P c1502216bb0c469abd5491603f914b60 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.738379 15258 raft_consensus.cc:1270] T 1a17e23216384550ac12b7b725f82b5a P e7ff2606284a4b339d7d9b7e74256850 [term 1 FOLLOWER]: Refusing update from remote peer 830d0fd8356b411e9ad8df94d9c9ab05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:43.738662 15663 consensus_queue.cc:1035] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7ff2606284a4b339d7d9b7e74256850" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:43.740186 15663 consensus_queue.cc:1035] T 1a17e23216384550ac12b7b725f82b5a P 830d0fd8356b411e9ad8df94d9c9ab05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1502216bb0c469abd5491603f914b60" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 45113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 26.9886 ms
time per row: 0.0134943 ms
I20250110 03:06:43.875515 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:43.875530 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:43.875576 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:43.882813 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:43.885645 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:43.885747 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:43.934738 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:43.934778 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:43.935357 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:43.940616 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:43.942732 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:43.943467 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:44.001291 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.001560 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.002651 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.002732 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.007614 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:44.010669 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.056703 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.057049 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.057731 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.057739 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.066363 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:44.066416 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.114274 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.115633 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.117362 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.121575 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.121735 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.128953 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:44.170130 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.170459 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.171730 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.172703 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.180171 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:44.180729 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.223229 14974 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.226384 14711 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.226763 15239 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:06:44.228217 14843 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.230674 15107 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:44.235982 14578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:44.254905 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 14517
I20250110 03:06:44.262991 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 14648
I20250110 03:06:44.269201 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 14780
I20250110 03:06:44.274755 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 14913
I20250110 03:06:44.280145 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 15045
I20250110 03:06:44.285159 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 15177
I20250110 03:06:44.290388 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 14426
2025-01-10T03:06:44Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.IntraLocationNoConcurrency (9645 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (9645 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20250110 03:06:44.330386 14414 test_util.cc:274] Using random seed: 1651721100
I20250110 03:06:44.330938 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:06:44.330972 14414 ts_itest-base.cc:113] --------------
I20250110 03:06:44.330993 14414 ts_itest-base.cc:114] 5 tablet servers
I20250110 03:06:44.331009 14414 ts_itest-base.cc:115] 3 replicas per TS
I20250110 03:06:44.331035 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:06:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:06:44Z Disabled control of system clock
I20250110 03:06:44.338160 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:41695
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:41695
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20250110 03:06:44.414379 16338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:44.414593 16338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:44.414630 16338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:44.415903 16338 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:44.415949 16338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:06:44.415973 16338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:44.415993 16338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:06:44.416019 16338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:06:44.417506 16338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:41695
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:41695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.16338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:44.417740 16338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:44.417960 16338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:44.422498 16346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.423501 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.424314 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:44.425096 16338 server_base.cc:1034] running on GCE node
I20250110 03:06:44.425307 16338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:44.425531 16338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:44.426661 16338 hybrid_clock.cc:648] HybridClock initialized: now 1736478404426642 us; error 33 us; skew 500 ppm
I20250110 03:06:44.427770 16338 webserver.cc:458] Webserver started at http://127.14.19.190:34281/ using document root <none> and password file <none>
I20250110 03:06:44.427954 16338 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:44.428004 16338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:44.428115 16338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:44.429049 16338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9284d8c1444f40dcb43a5f1e707d289d"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.429344 16338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9284d8c1444f40dcb43a5f1e707d289d"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.430456 16338 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:06:44.431183 16352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.431330 16338 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.431401 16338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9284d8c1444f40dcb43a5f1e707d289d"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.431460 16338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:44.442158 16338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:44.442452 16338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:44.442605 16338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:44.446318 16338 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:41695
I20250110 03:06:44.446357 16404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:41695 every 8 connection(s)
I20250110 03:06:44.446727 16338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:06:44.447327 16405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:44.449570 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d: Bootstrap starting.
I20250110 03:06:44.450124 16405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:44.450309 16405 log.cc:826] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:44.450829 16405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d: No bootstrap required, opened a new log
I20250110 03:06:44.451973 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 16338
I20250110 03:06:44.452078 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:06:44.452409 16405 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9284d8c1444f40dcb43a5f1e707d289d" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41695 } }
I20250110 03:06:44.452526 16405 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:44.452570 16405 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9284d8c1444f40dcb43a5f1e707d289d, State: Initialized, Role: FOLLOWER
I20250110 03:06:44.452899 16405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9284d8c1444f40dcb43a5f1e707d289d" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41695 } }
I20250110 03:06:44.452989 16405 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:06:44.453013 16405 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:06:44.453050 16405 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:44.453670 16405 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9284d8c1444f40dcb43a5f1e707d289d" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41695 } }
I20250110 03:06:44.453771 16405 leader_election.cc:304] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9284d8c1444f40dcb43a5f1e707d289d; no voters:
I20250110 03:06:44.453925 16405 leader_election.cc:290] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:06:44.454170 16410 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:44.454205 16405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:06:44.454283 16410 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [term 1 LEADER]: Becoming Leader. State: Replica: 9284d8c1444f40dcb43a5f1e707d289d, State: Running, Role: LEADER
I20250110 03:06:44.454368 16410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9284d8c1444f40dcb43a5f1e707d289d" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41695 } }
I20250110 03:06:44.454672 16411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9284d8c1444f40dcb43a5f1e707d289d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9284d8c1444f40dcb43a5f1e707d289d" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41695 } } }
I20250110 03:06:44.454787 16411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [sys.catalog]: This master's current role is: LEADER
I20250110 03:06:44.454672 16412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9284d8c1444f40dcb43a5f1e707d289d. Latest consensus state: current_term: 1 leader_uuid: "9284d8c1444f40dcb43a5f1e707d289d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9284d8c1444f40dcb43a5f1e707d289d" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41695 } } }
I20250110 03:06:44.454957 16412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d [sys.catalog]: This master's current role is: LEADER
I20250110 03:06:44.455111 16418 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:06:44.455811 16418 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:06:44.457267 16418 catalog_manager.cc:1348] Generated new cluster ID: f735ad6fa7c74c39b8e3a1fb795b6026
I20250110 03:06:44.457319 16418 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:06:44.469201 16418 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:06:44.469916 16418 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:06:44.476153 16418 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 9284d8c1444f40dcb43a5f1e707d289d: Generated new TSK 0
I20250110 03:06:44.476401 16418 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:06:44.479444 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--builtin_ntp_servers=127.14.19.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:44.579756 16429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:44.579952 16429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:44.579981 16429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:44.581138 16429 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:44.581185 16429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:44.581223 16429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:06:44.582701 16429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--never_fsync=true
--heap_profile_path=/tmp/kudu.16429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:44.582974 16429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:44.583212 16429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:44.588732 16435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.589661 16434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.590354 16437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:44.591203 16429 server_base.cc:1034] running on GCE node
I20250110 03:06:44.591368 16429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:44.591589 16429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:44.592717 16429 hybrid_clock.cc:648] HybridClock initialized: now 1736478404592694 us; error 42 us; skew 500 ppm
I20250110 03:06:44.594172 16429 webserver.cc:458] Webserver started at http://127.14.19.129:34655/ using document root <none> and password file <none>
I20250110 03:06:44.594370 16429 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:44.594419 16429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:44.594527 16429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:44.595445 16429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "75dd4411e79a4e8f854fd9baa57180fc"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.595737 16429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "75dd4411e79a4e8f854fd9baa57180fc"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.596962 16429 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:06:44.597666 16443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.597829 16429 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:44.597900 16429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "75dd4411e79a4e8f854fd9baa57180fc"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.597958 16429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:44.615823 16429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:44.616075 16429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:44.616178 16429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:44.616394 16429 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:44.616744 16429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:44.616783 16429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.616817 16429 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:44.616858 16429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.623834 16429 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:41963
I20250110 03:06:44.623888 16556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:41963 every 8 connection(s)
I20250110 03:06:44.624203 16429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:06:44.624562 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 16429
I20250110 03:06:44.624641 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:06:44.625736 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--builtin_ntp_servers=127.14.19.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:44.636009 16557 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41695
I20250110 03:06:44.636137 16557 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:44.636385 16557 heartbeater.cc:510] Master 127.14.19.190:41695 requested a full tablet report, sending...
I20250110 03:06:44.636952 16369 ts_manager.cc:194] Registered new tserver with Master: 75dd4411e79a4e8f854fd9baa57180fc (127.14.19.129:41963)
I20250110 03:06:44.637892 16369 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:47805
W20250110 03:06:44.703114 16560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:44.703277 16560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:44.703305 16560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:44.704500 16560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:44.704540 16560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:44.704576 16560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:06:44.705909 16560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--never_fsync=true
--heap_profile_path=/tmp/kudu.16560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:44.706171 16560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:44.706383 16560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:44.711125 16565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.711607 16568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.712226 16566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:44.713347 16560 server_base.cc:1034] running on GCE node
I20250110 03:06:44.713526 16560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:44.713721 16560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:44.714843 16560 hybrid_clock.cc:648] HybridClock initialized: now 1736478404714827 us; error 31 us; skew 500 ppm
I20250110 03:06:44.716015 16560 webserver.cc:458] Webserver started at http://127.14.19.130:34707/ using document root <none> and password file <none>
I20250110 03:06:44.716209 16560 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:44.716276 16560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:44.716393 16560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:44.717259 16560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "11c55e846cbe4a94bf5cfcb6a9e57300"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.717557 16560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "11c55e846cbe4a94bf5cfcb6a9e57300"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.718667 16560 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:06:44.719276 16574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.719421 16560 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.719489 16560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "11c55e846cbe4a94bf5cfcb6a9e57300"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.719568 16560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:44.735867 16560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:44.736130 16560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:44.736246 16560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:44.736464 16560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:44.736765 16560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:44.736799 16560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.736835 16560 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:44.736856 16560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.742508 16560 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:34485
I20250110 03:06:44.742601 16687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:34485 every 8 connection(s)
I20250110 03:06:44.742853 16560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:06:44.747781 16688 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41695
I20250110 03:06:44.747869 16688 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:44.748030 16688 heartbeater.cc:510] Master 127.14.19.190:41695 requested a full tablet report, sending...
I20250110 03:06:44.748360 16369 ts_manager.cc:194] Registered new tserver with Master: 11c55e846cbe4a94bf5cfcb6a9e57300 (127.14.19.130:34485)
I20250110 03:06:44.748713 16369 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:56121
I20250110 03:06:44.750255 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 16560
I20250110 03:06:44.750320 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:06:44.751236 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--builtin_ntp_servers=127.14.19.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:44.823871 16691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:44.824043 16691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:44.824060 16691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:44.825348 16691 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:44.825403 16691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:44.825425 16691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:06:44.826900 16691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--never_fsync=true
--heap_profile_path=/tmp/kudu.16691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:44.827128 16691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:44.827368 16691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:44.832005 16699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.832644 16697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.833706 16696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:44.834592 16691 server_base.cc:1034] running on GCE node
I20250110 03:06:44.834767 16691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:44.834967 16691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:44.836090 16691 hybrid_clock.cc:648] HybridClock initialized: now 1736478404836074 us; error 35 us; skew 500 ppm
I20250110 03:06:44.843129 16691 webserver.cc:458] Webserver started at http://127.14.19.131:43533/ using document root <none> and password file <none>
I20250110 03:06:44.843345 16691 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:44.843395 16691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:44.843508 16691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:44.844429 16691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dd99deaff6a04ebaa973433cce1f82db"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.844722 16691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dd99deaff6a04ebaa973433cce1f82db"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.845960 16691 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20250110 03:06:44.846659 16705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.846840 16691 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.846928 16691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dd99deaff6a04ebaa973433cce1f82db"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.846978 16691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:44.857102 16691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:44.857334 16691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:44.857434 16691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:44.857600 16691 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:44.857864 16691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:44.857890 16691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.857913 16691 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:44.857928 16691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.863130 16691 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:45695
I20250110 03:06:44.863193 16818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:45695 every 8 connection(s)
I20250110 03:06:44.863471 16691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:06:44.865011 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 16691
I20250110 03:06:44.865088 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:06:44.866086 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--builtin_ntp_servers=127.14.19.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:44.868386 16819 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41695
I20250110 03:06:44.868471 16819 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:44.868642 16819 heartbeater.cc:510] Master 127.14.19.190:41695 requested a full tablet report, sending...
I20250110 03:06:44.869045 16369 ts_manager.cc:194] Registered new tserver with Master: dd99deaff6a04ebaa973433cce1f82db (127.14.19.131:45695)
I20250110 03:06:44.869408 16369 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:59893
W20250110 03:06:44.943068 16822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:44.943255 16822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:44.943284 16822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:44.944507 16822 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:44.944550 16822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:44.944586 16822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:06:44.945992 16822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--never_fsync=true
--heap_profile_path=/tmp/kudu.16822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:44.946226 16822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:44.946468 16822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:44.951296 16827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.951948 16828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:44.952975 16830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:44.953804 16822 server_base.cc:1034] running on GCE node
I20250110 03:06:44.953984 16822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:44.954180 16822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:44.955276 16822 hybrid_clock.cc:648] HybridClock initialized: now 1736478404955258 us; error 33 us; skew 500 ppm
I20250110 03:06:44.956528 16822 webserver.cc:458] Webserver started at http://127.14.19.132:36639/ using document root <none> and password file <none>
I20250110 03:06:44.956732 16822 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:44.956780 16822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:44.956877 16822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:44.957785 16822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b1f6233d436a4047b21d924c5997807e"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.958084 16822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b1f6233d436a4047b21d924c5997807e"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.959221 16822 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20250110 03:06:44.959770 16836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.959918 16822 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:44.959985 16822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b1f6233d436a4047b21d924c5997807e"
format_stamp: "Formatted at 2025-01-10 03:06:44 on dist-test-slave-nrvn"
I20250110 03:06:44.960038 16822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:44.970651 16822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:44.970890 16822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:44.970995 16822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:44.971174 16822 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:44.971464 16822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:44.971494 16822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.971527 16822 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:44.971556 16822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:44.976850 16822 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:40013
I20250110 03:06:44.976902 16949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:40013 every 8 connection(s)
I20250110 03:06:44.977207 16822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:06:44.980386 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 16822
I20250110 03:06:44.980454 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:06:44.981427 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:0
--local_ip_for_outbound_sockets=127.14.19.133
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--builtin_ntp_servers=127.14.19.148:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:44.981834 16950 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41695
I20250110 03:06:44.981938 16950 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:44.982169 16950 heartbeater.cc:510] Master 127.14.19.190:41695 requested a full tablet report, sending...
I20250110 03:06:44.982528 16369 ts_manager.cc:194] Registered new tserver with Master: b1f6233d436a4047b21d924c5997807e (127.14.19.132:40013)
I20250110 03:06:44.982961 16369 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:46939
W20250110 03:06:45.058763 16953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:45.058941 16953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:45.058960 16953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:45.060259 16953 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:45.060400 16953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:45.060451 16953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:06:45.062170 16953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41695
--never_fsync=true
--heap_profile_path=/tmp/kudu.16953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:45.062378 16953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:45.062635 16953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:45.067559 16961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:45.068220 16958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:45.069237 16959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:45.070098 16953 server_base.cc:1034] running on GCE node
I20250110 03:06:45.070250 16953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:45.070443 16953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:45.071569 16953 hybrid_clock.cc:648] HybridClock initialized: now 1736478405071559 us; error 32 us; skew 500 ppm
I20250110 03:06:45.072820 16953 webserver.cc:458] Webserver started at http://127.14.19.133:40963/ using document root <none> and password file <none>
I20250110 03:06:45.073012 16953 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:45.073061 16953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:45.073163 16953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:45.074059 16953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c913c2cc99ae4915b8c77b49c9de51f8"
format_stamp: "Formatted at 2025-01-10 03:06:45 on dist-test-slave-nrvn"
I20250110 03:06:45.074344 16953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c913c2cc99ae4915b8c77b49c9de51f8"
format_stamp: "Formatted at 2025-01-10 03:06:45 on dist-test-slave-nrvn"
I20250110 03:06:45.075497 16953 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20250110 03:06:45.076201 16967 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:45.076398 16953 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:45.076473 16953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c913c2cc99ae4915b8c77b49c9de51f8"
format_stamp: "Formatted at 2025-01-10 03:06:45 on dist-test-slave-nrvn"
I20250110 03:06:45.076534 16953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:45.088603 16953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:45.088866 16953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:45.088975 16953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:45.089155 16953 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:45.089440 16953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:45.089473 16953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:45.089506 16953 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:45.089534 16953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:45.095045 16953 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:42161
I20250110 03:06:45.095090 17080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:42161 every 8 connection(s)
I20250110 03:06:45.095387 16953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:06:45.096093 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 16953
I20250110 03:06:45.096179 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 03:06:45.100152 17081 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41695
I20250110 03:06:45.100248 17081 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:45.100427 17081 heartbeater.cc:510] Master 127.14.19.190:41695 requested a full tablet report, sending...
I20250110 03:06:45.100840 16369 ts_manager.cc:194] Registered new tserver with Master: c913c2cc99ae4915b8c77b49c9de51f8 (127.14.19.133:42161)
I20250110 03:06:45.101260 16369 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:35159
I20250110 03:06:45.108192 14414 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 03:06:45.113651 16369 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55134:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250110 03:06:45.121438 16473 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec21aa334d43d9966bb9b5bb35b9e6 (DEFAULT_TABLE table=TestTable [id=609bd3ac65384c9b8729f14d9b6aacad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:45.121438 16753 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec21aa334d43d9966bb9b5bb35b9e6 (DEFAULT_TABLE table=TestTable [id=609bd3ac65384c9b8729f14d9b6aacad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:45.121438 16884 tablet_service.cc:1467] Processing CreateTablet for tablet 12ec21aa334d43d9966bb9b5bb35b9e6 (DEFAULT_TABLE table=TestTable [id=609bd3ac65384c9b8729f14d9b6aacad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:45.121738 16753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12ec21aa334d43d9966bb9b5bb35b9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:45.121737 16884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12ec21aa334d43d9966bb9b5bb35b9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:45.121737 16473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12ec21aa334d43d9966bb9b5bb35b9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:45.123922 17098 tablet_bootstrap.cc:492] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: Bootstrap starting.
I20250110 03:06:45.124616 17098 tablet_bootstrap.cc:654] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:45.124591 17099 tablet_bootstrap.cc:492] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: Bootstrap starting.
I20250110 03:06:45.124989 17098 log.cc:826] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:45.125212 17099 tablet_bootstrap.cc:654] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:45.125490 17099 log.cc:826] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:45.125659 17098 tablet_bootstrap.cc:492] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: No bootstrap required, opened a new log
I20250110 03:06:45.125726 17098 ts_tablet_manager.cc:1397] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:45.125815 17100 tablet_bootstrap.cc:492] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: Bootstrap starting.
I20250110 03:06:45.126204 17099 tablet_bootstrap.cc:492] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: No bootstrap required, opened a new log
I20250110 03:06:45.126255 17099 ts_tablet_manager.cc:1397] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:45.126418 17100 tablet_bootstrap.cc:654] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:45.126724 17100 log.cc:826] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:45.127404 17100 tablet_bootstrap.cc:492] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: No bootstrap required, opened a new log
I20250110 03:06:45.127544 17100 ts_tablet_manager.cc:1397] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250110 03:06:45.128131 17098 raft_consensus.cc:357] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.128275 17098 raft_consensus.cc:383] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:45.128306 17098 raft_consensus.cc:738] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75dd4411e79a4e8f854fd9baa57180fc, State: Initialized, Role: FOLLOWER
I20250110 03:06:45.128435 17098 consensus_queue.cc:260] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.128563 17099 raft_consensus.cc:357] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.128687 17099 raft_consensus.cc:383] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:45.128692 17098 ts_tablet_manager.cc:1428] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:45.128752 17099 raft_consensus.cc:738] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd99deaff6a04ebaa973433cce1f82db, State: Initialized, Role: FOLLOWER
I20250110 03:06:45.128810 16557 heartbeater.cc:502] Master 127.14.19.190:41695 was elected leader, sending a full tablet report...
I20250110 03:06:45.128929 17099 consensus_queue.cc:260] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.129216 16819 heartbeater.cc:502] Master 127.14.19.190:41695 was elected leader, sending a full tablet report...
I20250110 03:06:45.129524 17099 ts_tablet_manager.cc:1428] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:45.129918 17100 raft_consensus.cc:357] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.130016 17100 raft_consensus.cc:383] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:45.130045 17100 raft_consensus.cc:738] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f6233d436a4047b21d924c5997807e, State: Initialized, Role: FOLLOWER
I20250110 03:06:45.130153 17100 consensus_queue.cc:260] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.130379 17100 ts_tablet_manager.cc:1428] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250110 03:06:45.130420 16950 heartbeater.cc:502] Master 127.14.19.190:41695 was elected leader, sending a full tablet report...
W20250110 03:06:45.227701 16951 tablet.cc:2367] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:45.345911 17104 raft_consensus.cc:491] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:45.346028 17104 raft_consensus.cc:513] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.346360 17104 leader_election.cc:290] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1f6233d436a4047b21d924c5997807e (127.14.19.132:40013), dd99deaff6a04ebaa973433cce1f82db (127.14.19.131:45695)
I20250110 03:06:45.349145 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec21aa334d43d9966bb9b5bb35b9e6" candidate_uuid: "75dd4411e79a4e8f854fd9baa57180fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f6233d436a4047b21d924c5997807e" is_pre_election: true
I20250110 03:06:45.349157 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec21aa334d43d9966bb9b5bb35b9e6" candidate_uuid: "75dd4411e79a4e8f854fd9baa57180fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd99deaff6a04ebaa973433cce1f82db" is_pre_election: true
I20250110 03:06:45.349287 16904 raft_consensus.cc:2463] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75dd4411e79a4e8f854fd9baa57180fc in term 0.
I20250110 03:06:45.349293 16773 raft_consensus.cc:2463] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75dd4411e79a4e8f854fd9baa57180fc in term 0.
I20250110 03:06:45.349514 16446 leader_election.cc:304] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75dd4411e79a4e8f854fd9baa57180fc, dd99deaff6a04ebaa973433cce1f82db; no voters:
I20250110 03:06:45.349634 17104 raft_consensus.cc:2798] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:45.349695 17104 raft_consensus.cc:491] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:45.349721 17104 raft_consensus.cc:3054] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:45.350381 17104 raft_consensus.cc:513] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.350494 17104 leader_election.cc:290] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [CANDIDATE]: Term 1 election: Requested vote from peers b1f6233d436a4047b21d924c5997807e (127.14.19.132:40013), dd99deaff6a04ebaa973433cce1f82db (127.14.19.131:45695)
I20250110 03:06:45.350674 16773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec21aa334d43d9966bb9b5bb35b9e6" candidate_uuid: "75dd4411e79a4e8f854fd9baa57180fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd99deaff6a04ebaa973433cce1f82db"
I20250110 03:06:45.350684 16904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12ec21aa334d43d9966bb9b5bb35b9e6" candidate_uuid: "75dd4411e79a4e8f854fd9baa57180fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f6233d436a4047b21d924c5997807e"
I20250110 03:06:45.350734 16773 raft_consensus.cc:3054] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:45.350749 16904 raft_consensus.cc:3054] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:45.351341 16773 raft_consensus.cc:2463] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75dd4411e79a4e8f854fd9baa57180fc in term 1.
I20250110 03:06:45.351352 16904 raft_consensus.cc:2463] T 12ec21aa334d43d9966bb9b5bb35b9e6 P b1f6233d436a4047b21d924c5997807e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75dd4411e79a4e8f854fd9baa57180fc in term 1.
I20250110 03:06:45.351500 16446 leader_election.cc:304] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75dd4411e79a4e8f854fd9baa57180fc, dd99deaff6a04ebaa973433cce1f82db; no voters:
I20250110 03:06:45.351596 17104 raft_consensus.cc:2798] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:45.351733 17104 raft_consensus.cc:695] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [term 1 LEADER]: Becoming Leader. State: Replica: 75dd4411e79a4e8f854fd9baa57180fc, State: Running, Role: LEADER
I20250110 03:06:45.351898 17104 consensus_queue.cc:237] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } }
I20250110 03:06:45.352728 16369 catalog_manager.cc:5526] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 75dd4411e79a4e8f854fd9baa57180fc (127.14.19.129). New cstate: current_term: 1 leader_uuid: "75dd4411e79a4e8f854fd9baa57180fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f6233d436a4047b21d924c5997807e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd99deaff6a04ebaa973433cce1f82db" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 45695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75dd4411e79a4e8f854fd9baa57180fc" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 41963 } health_report { overall_health: HEALTHY } } }
W20250110 03:06:45.364055 16820 tablet.cc:2367] T 12ec21aa334d43d9966bb9b5bb35b9e6 P dd99deaff6a04ebaa973433cce1f82db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:06:45.375684 16558 tablet.cc:2367] T 12ec21aa334d43d9966bb9b5bb35b9e6 P 75dd4411e79a4e8f854fd9baa57180fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:45.403296 14414 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 03:06:45.403831 14414 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 75dd4411e79a4e8f854fd9baa57180fc to finish bootstrapping
I20250110 03:06:45.407569 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 11c55e846cbe4a94bf5cfcb6a9e57300 to finish bootstrapping
I20250110 03:06:45.410975 14414 ts_itest-base.cc:233] Waiting for 1 tablets on tserver dd99deaff6a04ebaa973433cce1f82db to finish bootstrapping
I20250110 03:06:45.413688 14414 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b1f6233d436a4047b21d924c5997807e to finish bootstrapping
I20250110 03:06:45.416481 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c913c2cc99ae4915b8c77b49c9de51f8 to finish bootstrapping
I20250110 03:06:45.532547 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.533479 16473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:45.534624 17015 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.536926 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.537403 16884 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.577903 17015 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.578751 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.581377 16884 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.582453 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.586772 16473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:45.704018 16473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:45.704100 17015 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.704324 16884 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.704929 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.707620 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.746084 16473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:45.748368 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.748622 17015 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.749375 16688 heartbeater.cc:502] Master 127.14.19.190:41695 was elected leader, sending a full tablet report...
I20250110 03:06:45.752712 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.755342 16884 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.805754 16473 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:45.806185 16622 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.807526 17015 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.811049 16753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.813716 16884 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:45.826686 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 16429
I20250110 03:06:45.831837 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 16560
I20250110 03:06:45.836635 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 16691
I20250110 03:06:45.841290 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 16822
I20250110 03:06:45.845485 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 16953
I20250110 03:06:45.849792 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 16338
2025-01-10T03:06:45Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (1540 ms)
[----------] 1 test from RebalanceStartCriteriaTest (1540 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20250110 03:06:45.870731 14414 test_util.cc:274] Using random seed: 1653261440
I20250110 03:06:45.871213 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:06:45.871248 14414 ts_itest-base.cc:113] --------------
I20250110 03:06:45.871259 14414 ts_itest-base.cc:114] 7 tablet servers
I20250110 03:06:45.871269 14414 ts_itest-base.cc:115] 2 replicas per TS
I20250110 03:06:45.871279 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:06:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:06:45Z Disabled control of system clock
I20250110 03:06:45.877733 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:35275
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:35275
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20250110 03:06:45.953177 17284 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20250110 03:06:45.953369 17284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:45.953394 17284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:45.953420 17284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:45.954617 17284 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:45.954658 17284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:06:45.954679 17284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:45.954694 17284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:06:45.954712 17284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:06:45.956089 17284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:35275
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:45.956321 17284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:45.956561 17284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:45.961081 17292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:45.961740 17289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:45.962859 17290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:45.963651 17284 server_base.cc:1034] running on GCE node
I20250110 03:06:45.963860 17284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:45.964087 17284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:45.965221 17284 hybrid_clock.cc:648] HybridClock initialized: now 1736478405965205 us; error 31 us; skew 500 ppm
I20250110 03:06:45.966404 17284 webserver.cc:458] Webserver started at http://127.14.19.190:42089/ using document root <none> and password file <none>
I20250110 03:06:45.966636 17284 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:45.966691 17284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:45.966790 17284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:45.967711 17284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7c800037a95e4de6bd43d671ea55076b"
format_stamp: "Formatted at 2025-01-10 03:06:45 on dist-test-slave-nrvn"
I20250110 03:06:45.968075 17284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7c800037a95e4de6bd43d671ea55076b"
format_stamp: "Formatted at 2025-01-10 03:06:45 on dist-test-slave-nrvn"
I20250110 03:06:45.969262 17284 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20250110 03:06:45.969873 17298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:45.970018 17284 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:45.970096 17284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7c800037a95e4de6bd43d671ea55076b"
format_stamp: "Formatted at 2025-01-10 03:06:45 on dist-test-slave-nrvn"
I20250110 03:06:45.970149 17284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:45.986893 17284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:45.987227 17284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:45.987354 17284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:45.990873 17284 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:35275
I20250110 03:06:45.990903 17350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:35275 every 8 connection(s)
I20250110 03:06:45.991216 17284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:06:45.991789 17351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:45.994138 17351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b: Bootstrap starting.
I20250110 03:06:45.994743 17351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:45.994979 17351 log.cc:826] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:45.995537 17351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b: No bootstrap required, opened a new log
I20250110 03:06:45.996881 17351 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c800037a95e4de6bd43d671ea55076b" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35275 } }
I20250110 03:06:45.997004 17351 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:45.997026 17351 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c800037a95e4de6bd43d671ea55076b, State: Initialized, Role: FOLLOWER
I20250110 03:06:45.997152 17351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c800037a95e4de6bd43d671ea55076b" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35275 } }
I20250110 03:06:45.997210 17351 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:06:45.997231 17351 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:06:45.997278 17351 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:45.997771 17351 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c800037a95e4de6bd43d671ea55076b" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35275 } }
I20250110 03:06:45.997855 17351 leader_election.cc:304] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c800037a95e4de6bd43d671ea55076b; no voters:
I20250110 03:06:45.997996 17351 leader_election.cc:290] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:06:45.998035 17354 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:45.998171 17354 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [term 1 LEADER]: Becoming Leader. State: Replica: 7c800037a95e4de6bd43d671ea55076b, State: Running, Role: LEADER
I20250110 03:06:45.998205 17351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:06:45.998277 17354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c800037a95e4de6bd43d671ea55076b" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35275 } }
I20250110 03:06:45.998689 17356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c800037a95e4de6bd43d671ea55076b. Latest consensus state: current_term: 1 leader_uuid: "7c800037a95e4de6bd43d671ea55076b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c800037a95e4de6bd43d671ea55076b" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35275 } } }
I20250110 03:06:45.998864 17356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [sys.catalog]: This master's current role is: LEADER
I20250110 03:06:45.998874 17355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c800037a95e4de6bd43d671ea55076b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c800037a95e4de6bd43d671ea55076b" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35275 } } }
I20250110 03:06:45.999117 17355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b [sys.catalog]: This master's current role is: LEADER
I20250110 03:06:45.999289 17364 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:06:45.999761 17364 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:06:46.001112 17364 catalog_manager.cc:1348] Generated new cluster ID: c2992a850466413b991a40589c163950
I20250110 03:06:46.001152 17364 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:06:46.001407 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 17284
I20250110 03:06:46.001487 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:06:46.015599 17364 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:06:46.016052 17364 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:06:46.020694 17364 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7c800037a95e4de6bd43d671ea55076b: Generated new TSK 0
I20250110 03:06:46.020854 17364 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:06:46.022830 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:46.114847 17375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.115018 17375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.115036 17375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.116204 17375 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.116245 17375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.116267 17375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:06:46.117749 17375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.17375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.117987 17375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.118232 17375 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.122884 17381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.123860 17380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.124648 17383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.125413 17375 server_base.cc:1034] running on GCE node
I20250110 03:06:46.125610 17375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.125861 17375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.127017 17375 hybrid_clock.cc:648] HybridClock initialized: now 1736478406127005 us; error 28 us; skew 500 ppm
I20250110 03:06:46.128525 17375 webserver.cc:458] Webserver started at http://127.14.19.129:35693/ using document root <none> and password file <none>
I20250110 03:06:46.128751 17375 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.128811 17375 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.128916 17375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.130016 17375 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "92bcfebb99cc47bba372938b64a40f4b"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.130364 17375 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "92bcfebb99cc47bba372938b64a40f4b"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.131886 17375 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20250110 03:06:46.132699 17389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.132894 17375 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:46.132968 17375 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "92bcfebb99cc47bba372938b64a40f4b"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.133064 17375 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.147192 17375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.147614 17375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.147778 17375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.147980 17375 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.148275 17375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.148308 17375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.148340 17375 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.148370 17375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.153589 17375 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:42521
I20250110 03:06:46.153638 17502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:42521 every 8 connection(s)
I20250110 03:06:46.153940 17375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:06:46.157518 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 17375
I20250110 03:06:46.157629 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:06:46.159094 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:46.159981 17503 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.160080 17503 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.160269 17503 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.160725 17315 ts_manager.cc:194] Registered new tserver with Master: 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:46.161332 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:42761
W20250110 03:06:46.238658 17506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.238818 17506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.238837 17506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.240039 17506 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.240084 17506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.240106 17506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:06:46.241480 17506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.17506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.241647 17506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.241837 17506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.246448 17511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.247334 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.247967 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.248690 17506 server_base.cc:1034] running on GCE node
I20250110 03:06:46.248876 17506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.249089 17506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.250212 17506 hybrid_clock.cc:648] HybridClock initialized: now 1736478406250203 us; error 28 us; skew 500 ppm
I20250110 03:06:46.251295 17506 webserver.cc:458] Webserver started at http://127.14.19.130:41397/ using document root <none> and password file <none>
I20250110 03:06:46.251451 17506 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.251488 17506 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.251555 17506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.252365 17506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "74fcf6c6434b49d8a777aaeda45d971e"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.252626 17506 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "74fcf6c6434b49d8a777aaeda45d971e"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.253631 17506 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:06:46.254247 17520 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.254436 17506 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:46.254501 17506 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "74fcf6c6434b49d8a777aaeda45d971e"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.254570 17506 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.301319 17506 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.301545 17506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.301626 17506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.301800 17506 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.302125 17506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.302157 17506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.302177 17506 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.302191 17506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.307513 17506 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:35185
I20250110 03:06:46.307596 17633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:35185 every 8 connection(s)
I20250110 03:06:46.307862 17506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:06:46.312578 17634 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.312673 17634 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.312842 17634 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.313176 17315 ts_manager.cc:194] Registered new tserver with Master: 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:46.313506 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:48777
I20250110 03:06:46.315043 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 17506
I20250110 03:06:46.315133 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:06:46.316185 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:46.388857 17637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.389026 17637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.389055 17637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.390223 17637 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.390266 17637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.390300 17637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:06:46.391659 17637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.17637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.391876 17637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.392092 17637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.396965 17642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.397887 17645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.398748 17643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.399513 17637 server_base.cc:1034] running on GCE node
I20250110 03:06:46.399672 17637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.399889 17637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.401021 17637 hybrid_clock.cc:648] HybridClock initialized: now 1736478406401003 us; error 34 us; skew 500 ppm
I20250110 03:06:46.402079 17637 webserver.cc:458] Webserver started at http://127.14.19.131:34259/ using document root <none> and password file <none>
I20250110 03:06:46.402254 17637 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.402299 17637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.402391 17637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.403307 17637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0b345888171f4abcb611a51e27cbcd59"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.403594 17637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0b345888171f4abcb611a51e27cbcd59"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.404690 17637 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:06:46.405264 17651 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.405421 17637 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:46.405488 17637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0b345888171f4abcb611a51e27cbcd59"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.405545 17637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.417950 17637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.418164 17637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.418262 17637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.418421 17637 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.418783 17637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.418826 17637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.418861 17637 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.418888 17637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.424149 17637 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:46605
I20250110 03:06:46.424201 17764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:46605 every 8 connection(s)
I20250110 03:06:46.424449 17637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:06:46.428766 17765 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.428849 17765 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.428993 17765 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.429283 17315 ts_manager.cc:194] Registered new tserver with Master: 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:46.429620 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:60657
I20250110 03:06:46.430505 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 17637
I20250110 03:06:46.430609 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:06:46.431475 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:46.507769 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.507917 17768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.507934 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.509166 17768 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.509222 17768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.509241 17768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:06:46.510512 17768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.17768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.510722 17768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.510949 17768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.515543 17774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.516287 17773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.517015 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.518091 17768 server_base.cc:1034] running on GCE node
I20250110 03:06:46.518285 17768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.518479 17768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.519603 17768 hybrid_clock.cc:648] HybridClock initialized: now 1736478406519599 us; error 44 us; skew 500 ppm
I20250110 03:06:46.520674 17768 webserver.cc:458] Webserver started at http://127.14.19.132:35305/ using document root <none> and password file <none>
I20250110 03:06:46.520876 17768 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.520922 17768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.521013 17768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.521919 17768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b61b88548de84191b17de06cca3c435a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.522213 17768 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b61b88548de84191b17de06cca3c435a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.523437 17768 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20250110 03:06:46.524091 17782 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.524255 17768 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:46.524318 17768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b61b88548de84191b17de06cca3c435a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.524366 17768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.533226 17768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.533401 17768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.533481 17768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.533618 17768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.533881 17768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.533921 17768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.533939 17768 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.533950 17768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.539188 17768 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:38659
I20250110 03:06:46.539258 17895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:38659 every 8 connection(s)
I20250110 03:06:46.539498 17768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:06:46.543871 17896 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.543952 17896 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.544119 17896 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.544471 17315 ts_manager.cc:194] Registered new tserver with Master: b61b88548de84191b17de06cca3c435a (127.14.19.132:38659)
I20250110 03:06:46.544793 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:44883
I20250110 03:06:46.545334 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 17768
I20250110 03:06:46.545395 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:06:46.546270 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:0
--local_ip_for_outbound_sockets=127.14.19.133
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:46.619871 17899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.620029 17899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.620066 17899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.621244 17899 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.621282 17899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.621304 17899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:06:46.622663 17899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.17899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.622851 17899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.623041 17899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.627957 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.628845 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.629624 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.630334 17899 server_base.cc:1034] running on GCE node
I20250110 03:06:46.630478 17899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.630683 17899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.631790 17899 hybrid_clock.cc:648] HybridClock initialized: now 1736478406631771 us; error 33 us; skew 500 ppm
I20250110 03:06:46.633167 17899 webserver.cc:458] Webserver started at http://127.14.19.133:39437/ using document root <none> and password file <none>
I20250110 03:06:46.633366 17899 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.633416 17899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.633522 17899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.634379 17899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d1e6d3913b04475f8962dd6496fbea0a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.634688 17899 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d1e6d3913b04475f8962dd6496fbea0a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.635798 17899 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:46.636409 17913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.636565 17899 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.636637 17899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1e6d3913b04475f8962dd6496fbea0a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.636698 17899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.653084 17899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.653342 17899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.653448 17899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.653647 17899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.653966 17899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.654000 17899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.654033 17899 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.654060 17899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.659307 17899 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:39385
I20250110 03:06:46.659343 18026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:39385 every 8 connection(s)
I20250110 03:06:46.659631 17899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:06:46.660295 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 17899
I20250110 03:06:46.660403 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 03:06:46.661475 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:0
--local_ip_for_outbound_sockets=127.14.19.134
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:46.666509 18027 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.666643 18027 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.666850 18027 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.667232 17315 ts_manager.cc:194] Registered new tserver with Master: d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385)
I20250110 03:06:46.667611 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:55563
W20250110 03:06:46.736228 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.736452 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.736486 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.737751 18030 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.737797 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.737818 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:06:46.739195 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.18030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.739393 18030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.739619 18030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.744479 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.745232 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.745913 18038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.747057 18030 server_base.cc:1034] running on GCE node
I20250110 03:06:46.747236 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.747445 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.748584 18030 hybrid_clock.cc:648] HybridClock initialized: now 1736478406748564 us; error 38 us; skew 500 ppm
I20250110 03:06:46.749745 18030 webserver.cc:458] Webserver started at http://127.14.19.134:42573/ using document root <none> and password file <none>
I20250110 03:06:46.749930 18030 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.749991 18030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.750097 18030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.751008 18030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4ce3080cc7de4c729423b95fb7158a07"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.751302 18030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4ce3080cc7de4c729423b95fb7158a07"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.752413 18030 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:06:46.753054 18044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.753211 18030 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:46.753278 18030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4ce3080cc7de4c729423b95fb7158a07"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.753334 18030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.771742 18030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.772030 18030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.772140 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.772351 18030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.772683 18030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.772717 18030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.772751 18030 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.772783 18030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.778097 18030 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:36693
I20250110 03:06:46.778163 18157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:36693 every 8 connection(s)
I20250110 03:06:46.778443 18030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:06:46.782907 18158 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.783012 18158 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.783210 18158 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.783551 17315 ts_manager.cc:194] Registered new tserver with Master: 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
I20250110 03:06:46.783910 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:48563
I20250110 03:06:46.785704 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 18030
I20250110 03:06:46.785774 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250110 03:06:46.786760 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.135:0
--local_ip_for_outbound_sockets=127.14.19.135
--webserver_interface=127.14.19.135
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:46.860478 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:46.860635 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:46.860652 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:46.861774 18161 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:46.861820 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:46.861842 18161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.135
I20250110 03:06:46.863356 18161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.19.135
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.18161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.135
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:46.863556 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:46.863770 18161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:46.868302 18166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.869136 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:46.869810 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:46.870491 18161 server_base.cc:1034] running on GCE node
I20250110 03:06:46.870658 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:46.870848 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:46.871982 18161 hybrid_clock.cc:648] HybridClock initialized: now 1736478406871971 us; error 27 us; skew 500 ppm
I20250110 03:06:46.873126 18161 webserver.cc:458] Webserver started at http://127.14.19.135:33191/ using document root <none> and password file <none>
I20250110 03:06:46.873303 18161 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:46.873346 18161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:46.873435 18161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:06:46.874265 18161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d28dc7130e374d8486cc68018c03fd34"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.874531 18161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d28dc7130e374d8486cc68018c03fd34"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.875604 18161 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:46.876231 18175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.876377 18161 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:46.876462 18161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d28dc7130e374d8486cc68018c03fd34"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:46.876525 18161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:46.912465 18161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:46.912744 18161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:46.912859 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:46.913089 18161 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:46.913411 18161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:46.913443 18161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.913477 18161 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:46.913503 18161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:46.918628 18161 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.135:32779
I20250110 03:06:46.918668 18288 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.135:32779 every 8 connection(s)
I20250110 03:06:46.918972 18161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250110 03:06:46.921078 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 18161
I20250110 03:06:46.921144 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20250110 03:06:46.923542 18289 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:46.923625 18289 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:46.923838 18289 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:46.924187 17315 ts_manager.cc:194] Registered new tserver with Master: d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779)
I20250110 03:06:46.924521 17315 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.135:36647
I20250110 03:06:46.932832 14414 external_mini_cluster.cc:934] 7 TS(s) registered with all masters
I20250110 03:06:46.938055 17315 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60734:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250110 03:06:46.945340 17699 tablet_service.cc:1467] Processing CreateTablet for tablet 888f2d12f3254197a6a490e23016bd46 (DEFAULT_TABLE table=TestTable [id=eb9fad4b6aed41feba5d3435ea9d883f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:46.945665 17699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 888f2d12f3254197a6a490e23016bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:46.945938 18092 tablet_service.cc:1467] Processing CreateTablet for tablet 888f2d12f3254197a6a490e23016bd46 (DEFAULT_TABLE table=TestTable [id=eb9fad4b6aed41feba5d3435ea9d883f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:46.946169 18092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 888f2d12f3254197a6a490e23016bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:46.947971 18304 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:46.948359 18305 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap starting.
I20250110 03:06:46.948532 18304 tablet_bootstrap.cc:654] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:46.948774 18304 log.cc:826] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:46.948879 18305 tablet_bootstrap.cc:654] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:46.949158 18305 log.cc:826] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:46.949388 18304 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:46.949447 18304 ts_tablet_manager.cc:1397] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:46.949707 18305 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: No bootstrap required, opened a new log
I20250110 03:06:46.949752 18305 ts_tablet_manager.cc:1397] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:46.950976 18304 raft_consensus.cc:357] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:46.951079 18304 raft_consensus.cc:383] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:46.951107 18304 raft_consensus.cc:738] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:46.951205 18304 consensus_queue.cc:260] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:46.951280 18305 raft_consensus.cc:357] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:46.951398 18305 raft_consensus.cc:383] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:46.951444 18304 ts_tablet_manager.cc:1428] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 03:06:46.951467 17765 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:06:46.951426 18305 raft_consensus.cc:738] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Initialized, Role: FOLLOWER
I20250110 03:06:46.951784 18305 consensus_queue.cc:260] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:46.951990 18305 ts_tablet_manager.cc:1428] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 03:06:46.952030 18158 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
W20250110 03:06:47.028971 18159 tablet.cc:2367] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:47.110235 18309 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:47.110354 18309 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:47.110663 18309 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:47.113111 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888f2d12f3254197a6a490e23016bd46" candidate_uuid: "4ce3080cc7de4c729423b95fb7158a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:47.113235 17719 raft_consensus.cc:2463] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce3080cc7de4c729423b95fb7158a07 in term 0.
I20250110 03:06:47.113430 18046 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 4ce3080cc7de4c729423b95fb7158a07; no voters:
I20250110 03:06:47.113584 18309 raft_consensus.cc:2798] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:47.113680 18309 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:47.113749 18309 raft_consensus.cc:3054] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:47.114391 18309 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:47.114490 18309 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 1 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:47.114673 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888f2d12f3254197a6a490e23016bd46" candidate_uuid: "4ce3080cc7de4c729423b95fb7158a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:47.114737 17719 raft_consensus.cc:3054] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:47.115275 17719 raft_consensus.cc:2463] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce3080cc7de4c729423b95fb7158a07 in term 1.
I20250110 03:06:47.115428 18046 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 4ce3080cc7de4c729423b95fb7158a07; no voters:
I20250110 03:06:47.115530 18309 raft_consensus.cc:2798] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:47.115681 18309 raft_consensus.cc:695] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Running, Role: LEADER
I20250110 03:06:47.115775 18309 consensus_queue.cc:237] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:47.116508 17315 catalog_manager.cc:5526] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134). New cstate: current_term: 1 leader_uuid: "4ce3080cc7de4c729423b95fb7158a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:47.122931 14414 external_mini_cluster.cc:934] 7 TS(s) registered with all masters
I20250110 03:06:47.123315 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 92bcfebb99cc47bba372938b64a40f4b to finish bootstrapping
I20250110 03:06:47.126807 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 74fcf6c6434b49d8a777aaeda45d971e to finish bootstrapping
I20250110 03:06:47.130074 14414 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0b345888171f4abcb611a51e27cbcd59 to finish bootstrapping
I20250110 03:06:47.132884 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b61b88548de84191b17de06cca3c435a to finish bootstrapping
I20250110 03:06:47.135907 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d1e6d3913b04475f8962dd6496fbea0a to finish bootstrapping
I20250110 03:06:47.139448 14414 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4ce3080cc7de4c729423b95fb7158a07 to finish bootstrapping
I20250110 03:06:47.142022 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d28dc7130e374d8486cc68018c03fd34 to finish bootstrapping
I20250110 03:06:47.144939 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 17768
I20250110 03:06:47.149962 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 17899
I20250110 03:06:47.154281 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 18030
I20250110 03:06:47.158938 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 18161
I20250110 03:06:47.162142 17503 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
W20250110 03:06:47.175204 17766 tablet.cc:2367] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:47.314215 17634 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:06:49.058362 18318 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:49.058452 18318 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:49.058744 18318 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
W20250110 03:06:49.059130 17652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.19.134:36693: connect: Connection refused (error 111)
W20250110 03:06:49.059502 17652 leader_election.cc:336] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693): Network error: Client connection negotiation failed: client connection to 127.14.19.134:36693: connect: Connection refused (error 111)
I20250110 03:06:49.059557 17652 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59; no voters: 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:49.059677 18318 raft_consensus.cc:2743] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 03:06:50.663312 18321 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:50.663405 18321 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:50.663549 18321 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
W20250110 03:06:50.664131 17652 leader_election.cc:336] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693): Network error: Client connection negotiation failed: client connection to 127.14.19.134:36693: connect: Connection refused (error 111)
I20250110 03:06:50.664166 17652 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59; no voters: 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:50.664242 18321 raft_consensus.cc:2743] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 03:06:50.913712 17315 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60734:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:50.918764 17566 tablet_service.cc:1467] Processing CreateTablet for tablet fa58142d4f86446b99561d0a0fb9fc96 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:50.918763 17437 tablet_service.cc:1467] Processing CreateTablet for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:50.918771 17435 tablet_service.cc:1467] Processing CreateTablet for tablet fa58142d4f86446b99561d0a0fb9fc96 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:50.918771 17436 tablet_service.cc:1467] Processing CreateTablet for tablet 7906c51301244bd4a592d8e01e6ad602 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:50.919082 17437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:50.919082 17566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa58142d4f86446b99561d0a0fb9fc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:50.919173 17435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa58142d4f86446b99561d0a0fb9fc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:50.919246 17436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7906c51301244bd4a592d8e01e6ad602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:50.919327 17567 tablet_service.cc:1467] Processing CreateTablet for tablet 7906c51301244bd4a592d8e01e6ad602 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:50.919412 17567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7906c51301244bd4a592d8e01e6ad602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:50.921757 18333 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:50.922271 18333 tablet_bootstrap.cc:654] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:50.918840 17568 tablet_service.cc:1467] Processing CreateTablet for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 (DEFAULT_TABLE table=rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:50.922698 17568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:50.922741 18333 log.cc:826] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:50.923347 18333 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:50.923398 18333 ts_tablet_manager.cc:1397] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:50.923882 18332 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:50.924410 18332 tablet_bootstrap.cc:654] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:50.924674 18332 log.cc:826] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:50.925133 18333 raft_consensus.cc:357] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.925261 18333 raft_consensus.cc:383] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:50.925299 18333 raft_consensus.cc:738] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:50.925305 18332 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:50.925351 18332 ts_tablet_manager.cc:1397] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:50.925407 18333 consensus_queue.cc:260] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.925661 18333 ts_tablet_manager.cc:1428] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 03:06:50.925786 18333 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:50.926232 18333 tablet_bootstrap.cc:654] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:50.926904 18333 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:50.926956 18333 ts_tablet_manager.cc:1397] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:50.927101 18333 raft_consensus.cc:357] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.927163 18333 raft_consensus.cc:383] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:50.927199 18333 raft_consensus.cc:738] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:50.927268 18333 consensus_queue.cc:260] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.927352 18333 ts_tablet_manager.cc:1428] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:06:50.927450 18333 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:50.927489 18332 raft_consensus.cc:357] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.927615 18332 raft_consensus.cc:383] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:50.927690 18332 raft_consensus.cc:738] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:50.927801 18332 consensus_queue.cc:260] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.927920 18333 tablet_bootstrap.cc:654] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:50.928089 18332 ts_tablet_manager.cc:1428] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:06:50.928175 18332 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:50.928573 18332 tablet_bootstrap.cc:654] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:50.928756 18333 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:50.928846 18333 ts_tablet_manager.cc:1397] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:50.928978 18333 raft_consensus.cc:357] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.929050 18333 raft_consensus.cc:383] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:50.929080 18333 raft_consensus.cc:738] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:50.929143 18332 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:50.929138 18333 consensus_queue.cc:260] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.929183 18332 ts_tablet_manager.cc:1397] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:50.929244 18333 ts_tablet_manager.cc:1428] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:50.929324 18332 raft_consensus.cc:357] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.929404 18332 raft_consensus.cc:383] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:50.929433 18332 raft_consensus.cc:738] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:50.929481 18332 consensus_queue.cc:260] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:50.929595 18332 ts_tablet_manager.cc:1428] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:50.929653 18332 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:50.930027 18332 tablet_bootstrap.cc:654] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:50.930560 18332 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:50.930603 18332 ts_tablet_manager.cc:1397] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:50.930748 18332 raft_consensus.cc:357] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.930794 18332 raft_consensus.cc:383] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:50.930814 18332 raft_consensus.cc:738] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:50.930846 18332 consensus_queue.cc:260] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.930931 18332 ts_tablet_manager.cc:1428] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:50.949654 18336 raft_consensus.cc:491] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:50.949721 18336 raft_consensus.cc:513] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.949913 18336 leader_election.cc:290] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:50.952225 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d2ceb09bc5442f8c329ca6b3ad6f35" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e" is_pre_election: true
I20250110 03:06:50.952314 17588 raft_consensus.cc:2463] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 0.
I20250110 03:06:50.952478 17391 leader_election.cc:304] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:50.952637 18336 raft_consensus.cc:2798] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:50.952700 18336 raft_consensus.cc:491] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:50.952729 18336 raft_consensus.cc:3054] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:50.953255 18336 raft_consensus.cc:513] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.953342 18336 leader_election.cc:290] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:50.953526 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83d2ceb09bc5442f8c329ca6b3ad6f35" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:50.953588 17588 raft_consensus.cc:3054] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:50.954134 17588 raft_consensus.cc:2463] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 1.
I20250110 03:06:50.954298 17391 leader_election.cc:304] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:50.954391 18336 raft_consensus.cc:2798] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:50.954516 18336 raft_consensus.cc:695] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Leader. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:06:50.954603 18336 consensus_queue.cc:237] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:50.955138 17315 catalog_manager.cc:5526] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129). New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } }
W20250110 03:06:51.061008 17635 tablet.cc:2367] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:06:51.157748 17504 tablet.cc:2367] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:51.196388 18338 raft_consensus.cc:491] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.196478 18338 raft_consensus.cc:513] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.196800 18338 leader_election.cc:290] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:51.199425 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa58142d4f86446b99561d0a0fb9fc96" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b" is_pre_election: true
I20250110 03:06:51.199527 17457 raft_consensus.cc:2463] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 0.
I20250110 03:06:51.199725 17521 leader_election.cc:304] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.199869 18338 raft_consensus.cc:2798] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.199934 18338 raft_consensus.cc:491] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.199959 18338 raft_consensus.cc:3054] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.200588 18338 raft_consensus.cc:513] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.200700 18338 leader_election.cc:290] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:51.200875 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa58142d4f86446b99561d0a0fb9fc96" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:51.200935 17457 raft_consensus.cc:3054] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.201424 17457 raft_consensus.cc:2463] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 1.
I20250110 03:06:51.201575 17521 leader_election.cc:304] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.201694 18338 raft_consensus.cc:2798] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.201817 18338 raft_consensus.cc:695] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:51.201917 18338 consensus_queue.cc:237] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.202632 17315 catalog_manager.cc:5526] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:51.214120 18338 raft_consensus.cc:491] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.214185 18338 raft_consensus.cc:513] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.214301 18338 leader_election.cc:290] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:51.214478 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b" is_pre_election: true
I20250110 03:06:51.214558 17457 raft_consensus.cc:2463] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 0.
I20250110 03:06:51.214700 17521 leader_election.cc:304] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.214823 18338 raft_consensus.cc:2798] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.214886 18338 raft_consensus.cc:491] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.214922 18338 raft_consensus.cc:3054] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.215438 18338 raft_consensus.cc:513] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.215529 18338 leader_election.cc:290] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:51.215672 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:51.215734 17457 raft_consensus.cc:3054] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.216209 17457 raft_consensus.cc:2463] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 1.
I20250110 03:06:51.216353 17521 leader_election.cc:304] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.216429 18338 raft_consensus.cc:2798] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.216472 18338 raft_consensus.cc:695] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:51.216504 18338 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.217024 17315 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } }
W20250110 03:06:51.338641 18349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:51.338793 18349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:51.340020 18349 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250110 03:06:51.376196 17457 raft_consensus.cc:1270] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:51.376209 17456 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:51.376632 18346 consensus_queue.cc:1035] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:51.376718 17588 raft_consensus.cc:1270] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:51.376837 18346 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:51.377072 18336 consensus_queue.cc:1035] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:51.378916 18378 mvcc.cc:204] Tried to move back new op lower bound from 7112615572994424832 to 7112615572283453440. Current Snapshot: MvccSnapshot[applied={T|T < 7112615572994424832}]
I20250110 03:06:51.379400 18381 mvcc.cc:204] Tried to move back new op lower bound from 7112615572994424832 to 7112615572283453440. Current Snapshot: MvccSnapshot[applied={T|T < 7112615572994424832}]
INSERT report
rows total: 2000
time total: 20.3178 ms
time per row: 0.0101589 ms
I20250110 03:06:51.388772 17309 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60734:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:51.390038 17437 tablet_service.cc:1467] Processing CreateTablet for tablet e365416b475a4288b17d235deddf139d (DEFAULT_TABLE table=rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.390164 17437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e365416b475a4288b17d235deddf139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.390147 17698 tablet_service.cc:1467] Processing CreateTablet for tablet 701c561086ce457e883794c11cb34ab6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.390193 17697 tablet_service.cc:1467] Processing CreateTablet for tablet f8b5177de1fa4e1c9668453452f2064f (DEFAULT_TABLE table=rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.390303 17698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c561086ce457e883794c11cb34ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.390324 17568 tablet_service.cc:1467] Processing CreateTablet for tablet 701c561086ce457e883794c11cb34ab6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.390147 17699 tablet_service.cc:1467] Processing CreateTablet for tablet e365416b475a4288b17d235deddf139d (DEFAULT_TABLE table=rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.390414 17568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c561086ce457e883794c11cb34ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.390437 17699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e365416b475a4288b17d235deddf139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.391470 17697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8b5177de1fa4e1c9668453452f2064f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.391556 18332 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:51.391894 18332 tablet_bootstrap.cc:654] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.392352 18333 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:51.392467 18393 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:51.392652 18333 tablet_bootstrap.cc:654] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.392868 18393 tablet_bootstrap.cc:654] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.393040 17437 tablet_service.cc:1467] Processing CreateTablet for tablet f8b5177de1fa4e1c9668453452f2064f (DEFAULT_TABLE table=rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.393136 17437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8b5177de1fa4e1c9668453452f2064f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.393173 18333 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:51.393229 18333 ts_tablet_manager.cc:1397] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:51.393357 18333 raft_consensus.cc:357] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.393426 18333 raft_consensus.cc:383] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.393452 18333 raft_consensus.cc:738] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.393512 18393 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:51.393498 18333 consensus_queue.cc:260] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.393585 18393 ts_tablet_manager.cc:1397] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:51.393595 18333 ts_tablet_manager.cc:1428] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.393761 18393 raft_consensus.cc:357] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.393810 18393 raft_consensus.cc:383] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.393829 18393 raft_consensus.cc:738] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.393885 18393 consensus_queue.cc:260] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.394025 18393 ts_tablet_manager.cc:1428] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.394076 18333 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:51.394107 18393 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:51.394698 18333 tablet_bootstrap.cc:654] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.394719 18332 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:51.394740 18393 tablet_bootstrap.cc:654] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.394757 18332 ts_tablet_manager.cc:1397] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:51.394887 18332 raft_consensus.cc:357] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.394933 18332 raft_consensus.cc:383] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.394946 18332 raft_consensus.cc:738] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.394984 18332 consensus_queue.cc:260] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.395066 18332 ts_tablet_manager.cc:1428] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.395339 18333 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:51.395386 18333 ts_tablet_manager.cc:1397] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:51.395512 18333 raft_consensus.cc:357] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.395552 18333 raft_consensus.cc:383] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.395591 18333 raft_consensus.cc:738] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.395643 18333 consensus_queue.cc:260] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.395757 18333 ts_tablet_manager.cc:1428] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.395807 18393 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:51.395850 18393 ts_tablet_manager.cc:1397] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:06:51.396186 18393 raft_consensus.cc:357] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.396245 18393 raft_consensus.cc:383] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.396265 18393 raft_consensus.cc:738] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.396328 18393 consensus_queue.cc:260] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.396425 18393 ts_tablet_manager.cc:1428] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.396488 18393 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:51.396857 18393 tablet_bootstrap.cc:654] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.397389 18393 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:51.397436 18393 ts_tablet_manager.cc:1397] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:51.397567 18393 raft_consensus.cc:357] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.397627 18393 raft_consensus.cc:383] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.397650 18393 raft_consensus.cc:738] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.397718 18393 consensus_queue.cc:260] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.397800 18393 ts_tablet_manager.cc:1428] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.413935 18397 raft_consensus.cc:491] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.413995 18397 raft_consensus.cc:513] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.414126 18397 leader_election.cc:290] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:51.416301 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701c561086ce457e883794c11cb34ab6" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e" is_pre_election: true
I20250110 03:06:51.416364 17588 raft_consensus.cc:2463] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 0.
I20250110 03:06:51.416501 17653 leader_election.cc:304] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:51.416591 18397 raft_consensus.cc:2798] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.416653 18397 raft_consensus.cc:491] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.416683 18397 raft_consensus.cc:3054] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.417196 18397 raft_consensus.cc:513] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.417287 18397 leader_election.cc:290] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:51.417431 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701c561086ce457e883794c11cb34ab6" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:51.417484 17588 raft_consensus.cc:3054] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.417949 17588 raft_consensus.cc:2463] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 1.
I20250110 03:06:51.418129 17653 leader_election.cc:304] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:51.418208 18397 raft_consensus.cc:2798] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.418314 18397 raft_consensus.cc:695] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Becoming Leader. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:51.418380 18397 consensus_queue.cc:237] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.418987 17309 catalog_manager.cc:5526] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:51.474690 18343 raft_consensus.cc:491] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.474766 18343 raft_consensus.cc:513] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.474892 18343 leader_election.cc:290] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:51.477182 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365416b475a4288b17d235deddf139d" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:51.477259 17719 raft_consensus.cc:2463] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 0.
I20250110 03:06:51.477403 17391 leader_election.cc:304] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.477494 18343 raft_consensus.cc:2798] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.477547 18343 raft_consensus.cc:491] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.477572 18343 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.478173 18343 raft_consensus.cc:513] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.478272 18343 leader_election.cc:290] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:51.478421 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365416b475a4288b17d235deddf139d" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:51.478477 17719 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.478967 17719 raft_consensus.cc:2463] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 1.
I20250110 03:06:51.479120 17391 leader_election.cc:304] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.479200 18343 raft_consensus.cc:2798] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.479260 18343 raft_consensus.cc:695] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Leader. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:06:51.479310 18343 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.479789 17309 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129). New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:51.627135 18343 raft_consensus.cc:491] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.627228 18343 raft_consensus.cc:513] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.627377 18343 leader_election.cc:290] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:51.627589 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b5177de1fa4e1c9668453452f2064f" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:51.627689 17719 raft_consensus.cc:2463] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 0.
I20250110 03:06:51.627841 17391 leader_election.cc:304] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.627960 18343 raft_consensus.cc:2798] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.628021 18343 raft_consensus.cc:491] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.628042 18343 raft_consensus.cc:3054] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.628620 18343 raft_consensus.cc:513] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.628719 18343 leader_election.cc:290] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:51.628875 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8b5177de1fa4e1c9668453452f2064f" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:51.628940 17719 raft_consensus.cc:3054] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.629402 17719 raft_consensus.cc:2463] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 1.
I20250110 03:06:51.629546 17391 leader_election.cc:304] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:51.629624 18343 raft_consensus.cc:2798] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.629683 18343 raft_consensus.cc:695] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Leader. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:06:51.629739 18343 consensus_queue.cc:237] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:51.630213 17309 catalog_manager.cc:5526] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129). New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } }
W20250110 03:06:51.745679 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:51.745855 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:51.747099 18408 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250110 03:06:51.777782 17719 raft_consensus.cc:1270] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:51.777799 17718 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:06:51.777858 17588 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:51.778103 18397 consensus_queue.cc:1035] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:51.778359 18436 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:51.778590 18436 consensus_queue.cc:1035] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:51.780758 18440 mvcc.cc:204] Tried to move back new op lower bound from 7112615574639452160 to 7112615573170036736. Current Snapshot: MvccSnapshot[applied={T|T < 7112615574639452160}]
INSERT report
rows total: 2000
time total: 21.2146 ms
time per row: 0.0106073 ms
I20250110 03:06:51.790624 17314 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60734:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:51.791932 17699 tablet_service.cc:1467] Processing CreateTablet for tablet 8dec030e45a84eb89321f275d0013826 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.792011 17697 tablet_service.cc:1467] Processing CreateTablet for tablet c4d68bc455bd4920ba99280f593678ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.791996 17437 tablet_service.cc:1467] Processing CreateTablet for tablet f3ec6ddcb58d48d89f4cea0b3db80035 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.792059 17699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dec030e45a84eb89321f275d0013826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.792052 17568 tablet_service.cc:1467] Processing CreateTablet for tablet 8dec030e45a84eb89321f275d0013826 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.792114 17437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ec6ddcb58d48d89f4cea0b3db80035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.792145 17567 tablet_service.cc:1467] Processing CreateTablet for tablet c4d68bc455bd4920ba99280f593678ec (DEFAULT_TABLE table=rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.792146 17568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dec030e45a84eb89321f275d0013826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.792246 17567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d68bc455bd4920ba99280f593678ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.793342 18333 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:51.793349 18332 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:51.793368 17566 tablet_service.cc:1467] Processing CreateTablet for tablet f3ec6ddcb58d48d89f4cea0b3db80035 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:51.793444 17566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ec6ddcb58d48d89f4cea0b3db80035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.793669 18332 tablet_bootstrap.cc:654] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.793740 18333 tablet_bootstrap.cc:654] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.794194 18332 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:51.794246 18332 ts_tablet_manager.cc:1397] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:51.794363 18332 raft_consensus.cc:357] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.794418 18332 raft_consensus.cc:383] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.794432 18332 raft_consensus.cc:738] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.794467 18332 consensus_queue.cc:260] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.794521 18393 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:51.794585 18332 ts_tablet_manager.cc:1428] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.794644 18332 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:51.794843 18393 tablet_bootstrap.cc:654] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.795019 18332 tablet_bootstrap.cc:654] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.795190 17697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d68bc455bd4920ba99280f593678ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:51.795730 18332 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:51.795770 18332 ts_tablet_manager.cc:1397] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:51.795892 18332 raft_consensus.cc:357] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.795934 18332 raft_consensus.cc:383] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.795948 18332 raft_consensus.cc:738] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.795984 18332 consensus_queue.cc:260] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.796046 18332 ts_tablet_manager.cc:1428] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.796087 18332 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:51.796351 18393 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:51.796403 18393 ts_tablet_manager.cc:1397] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:51.796449 18332 tablet_bootstrap.cc:654] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.796541 18393 raft_consensus.cc:357] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.796587 18393 raft_consensus.cc:383] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.796600 18393 raft_consensus.cc:738] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.796633 18393 consensus_queue.cc:260] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.796731 18393 ts_tablet_manager.cc:1428] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.796814 18393 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:51.797024 18332 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:51.797063 18332 ts_tablet_manager.cc:1397] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:51.797166 18393 tablet_bootstrap.cc:654] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:51.797188 18332 raft_consensus.cc:357] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.797235 18332 raft_consensus.cc:383] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.797248 18332 raft_consensus.cc:738] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.797281 18332 consensus_queue.cc:260] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.797365 18332 ts_tablet_manager.cc:1428] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.797861 18393 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:51.797911 18393 ts_tablet_manager.cc:1397] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:51.798049 18393 raft_consensus.cc:357] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.798108 18393 raft_consensus.cc:383] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.798126 18393 raft_consensus.cc:738] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.798162 18393 consensus_queue.cc:260] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.798256 18393 ts_tablet_manager.cc:1428] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.798664 18333 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:51.798717 18333 ts_tablet_manager.cc:1397] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250110 03:06:51.798950 18333 raft_consensus.cc:357] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.799058 18333 raft_consensus.cc:383] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:51.799089 18333 raft_consensus.cc:738] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:51.799135 18333 consensus_queue.cc:260] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.799224 18333 ts_tablet_manager.cc:1428] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:51.830422 18402 raft_consensus.cc:491] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.830502 18402 raft_consensus.cc:513] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.830675 18402 leader_election.cc:290] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:51.830849 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e" is_pre_election: true
I20250110 03:06:51.830928 17588 raft_consensus.cc:2463] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 0.
I20250110 03:06:51.831068 17653 leader_election.cc:304] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:51.831151 18402 raft_consensus.cc:2798] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.831193 18402 raft_consensus.cc:491] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.831223 18402 raft_consensus.cc:3054] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.831948 18402 raft_consensus.cc:513] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.832049 18402 leader_election.cc:290] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:51.832190 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:51.832249 17588 raft_consensus.cc:3054] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.832731 17588 raft_consensus.cc:2463] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 1.
I20250110 03:06:51.832880 17653 leader_election.cc:304] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:51.832957 18402 raft_consensus.cc:2798] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.833009 18402 raft_consensus.cc:695] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Becoming Leader. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:51.833041 18402 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:51.833525 17314 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:51.857959 18402 raft_consensus.cc:491] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:51.858037 18402 raft_consensus.cc:513] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.858151 18402 leader_election.cc:290] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:51.858354 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dec030e45a84eb89321f275d0013826" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e" is_pre_election: true
I20250110 03:06:51.858436 17588 raft_consensus.cc:2463] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 0.
I20250110 03:06:51.858583 17653 leader_election.cc:304] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:51.858678 18402 raft_consensus.cc:2798] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:51.858716 18402 raft_consensus.cc:491] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:51.858743 18402 raft_consensus.cc:3054] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.859238 18402 raft_consensus.cc:513] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.859316 18402 leader_election.cc:290] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:51.859522 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dec030e45a84eb89321f275d0013826" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:51.859589 17588 raft_consensus.cc:3054] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:51.860066 17588 raft_consensus.cc:2463] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 1.
I20250110 03:06:51.860222 17653 leader_election.cc:304] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:51.860308 18402 raft_consensus.cc:2798] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:51.860353 18402 raft_consensus.cc:695] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Becoming Leader. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:51.860406 18402 consensus_queue.cc:237] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:51.860924 17314 catalog_manager.cc:5526] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:52.240859 18402 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:52.267513 18437 raft_consensus.cc:491] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:52.267596 18437 raft_consensus.cc:513] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:52.267740 18437 leader_election.cc:290] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:52.267897 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ec6ddcb58d48d89f4cea0b3db80035" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e" is_pre_election: true
I20250110 03:06:52.267989 17588 raft_consensus.cc:2463] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 0.
I20250110 03:06:52.268149 17391 leader_election.cc:304] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.268244 18437 raft_consensus.cc:2798] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:52.268296 18437 raft_consensus.cc:491] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:52.268319 18437 raft_consensus.cc:3054] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.268925 18437 raft_consensus.cc:513] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:52.269034 18437 leader_election.cc:290] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:52.269181 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ec6ddcb58d48d89f4cea0b3db80035" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:52.269239 17588 raft_consensus.cc:3054] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.269702 17588 raft_consensus.cc:2463] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 1.
I20250110 03:06:52.269847 17391 leader_election.cc:304] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.269924 18437 raft_consensus.cc:2798] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:52.269975 18437 raft_consensus.cc:695] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Leader. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:06:52.270020 18437 consensus_queue.cc:237] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:52.270529 17312 catalog_manager.cc:5526] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129). New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:52.417872 18456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:52.418036 18456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:52.419276 18456 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20250110 03:06:52.439502 18402 consensus_queue.cc:1035] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:52.449412 17588 raft_consensus.cc:1270] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:52.449644 18437 consensus_queue.cc:1035] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 18.9005 ms
time per row: 0.00945025 ms
I20250110 03:06:52.462468 17312 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60734:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:52.463982 17566 tablet_service.cc:1467] Processing CreateTablet for tablet c8b6b5410b4e4405b419e22daf379c6a (DEFAULT_TABLE table=rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:52.463997 17697 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2221475806475c83aa9867a548f51f (DEFAULT_TABLE table=rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:52.464021 17699 tablet_service.cc:1467] Processing CreateTablet for tablet 593fb2f6a0a7417ca54b800249318eae (DEFAULT_TABLE table=rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:52.464113 17697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2221475806475c83aa9867a548f51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:52.464113 17566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b6b5410b4e4405b419e22daf379c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:52.464092 17437 tablet_service.cc:1467] Processing CreateTablet for tablet 8b2221475806475c83aa9867a548f51f (DEFAULT_TABLE table=rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:52.464206 17437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2221475806475c83aa9867a548f51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:52.464222 17699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593fb2f6a0a7417ca54b800249318eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:52.465438 17435 tablet_service.cc:1467] Processing CreateTablet for tablet 593fb2f6a0a7417ca54b800249318eae (DEFAULT_TABLE table=rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:52.465482 18504 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:52.465571 17436 tablet_service.cc:1467] Processing CreateTablet for tablet c8b6b5410b4e4405b419e22daf379c6a (DEFAULT_TABLE table=rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:52.465673 17436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b6b5410b4e4405b419e22daf379c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:52.465844 18504 tablet_bootstrap.cc:654] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:52.466358 18504 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:52.466399 18504 ts_tablet_manager.cc:1397] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:52.466526 18504 raft_consensus.cc:357] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.466616 18504 raft_consensus.cc:383] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:52.466621 17435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593fb2f6a0a7417ca54b800249318eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:52.466761 18502 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:52.466765 18504 raft_consensus.cc:738] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:52.466863 18504 consensus_queue.cc:260] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.467038 18504 ts_tablet_manager.cc:1428] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:52.467094 18504 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:52.467200 18502 tablet_bootstrap.cc:654] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:52.467438 18504 tablet_bootstrap.cc:654] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:52.467555 18503 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:52.467991 18504 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:52.467995 18503 tablet_bootstrap.cc:654] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:52.468057 18504 ts_tablet_manager.cc:1397] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:52.468207 18504 raft_consensus.cc:357] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.468237 18502 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:52.468274 18504 raft_consensus.cc:383] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:52.468284 18502 ts_tablet_manager.cc:1397] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:52.468299 18504 raft_consensus.cc:738] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:52.468354 18504 consensus_queue.cc:260] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.468441 18504 ts_tablet_manager.cc:1428] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:52.468500 18504 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:52.468422 18502 raft_consensus.cc:357] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.468576 18502 raft_consensus.cc:383] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:52.468621 18502 raft_consensus.cc:738] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:52.468680 18502 consensus_queue.cc:260] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.468806 18502 ts_tablet_manager.cc:1428] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:52.468876 18504 tablet_bootstrap.cc:654] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:52.468878 18503 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:52.468894 18502 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:52.468994 18503 ts_tablet_manager.cc:1397] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:52.469179 18503 raft_consensus.cc:357] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.469274 18503 raft_consensus.cc:383] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:52.469308 18503 raft_consensus.cc:738] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:52.469370 18503 consensus_queue.cc:260] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.469416 18502 tablet_bootstrap.cc:654] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:52.469488 18503 ts_tablet_manager.cc:1428] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:52.469969 18504 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:52.470013 18504 ts_tablet_manager.cc:1397] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:52.470119 18502 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:52.470125 18504 raft_consensus.cc:357] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.470155 18502 ts_tablet_manager.cc:1397] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:52.470177 18504 raft_consensus.cc:383] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:52.470196 18504 raft_consensus.cc:738] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:52.470242 18504 consensus_queue.cc:260] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.470317 18504 ts_tablet_manager.cc:1428] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:52.470296 18502 raft_consensus.cc:357] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.470347 18502 raft_consensus.cc:383] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:52.470369 18502 raft_consensus.cc:738] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:52.470410 18502 consensus_queue.cc:260] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.470489 18502 ts_tablet_manager.cc:1428] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:52.501461 18437 raft_consensus.cc:491] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:52.501534 18437 raft_consensus.cc:513] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.501647 18437 leader_election.cc:290] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:52.501854 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:52.501936 17718 raft_consensus.cc:2463] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 0.
I20250110 03:06:52.502089 17391 leader_election.cc:304] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.502174 18437 raft_consensus.cc:2798] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:52.502218 18437 raft_consensus.cc:491] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:52.502239 18437 raft_consensus.cc:3054] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.502825 18437 raft_consensus.cc:513] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.502919 18437 leader_election.cc:290] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:52.503085 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:52.503139 17718 raft_consensus.cc:3054] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.503722 17718 raft_consensus.cc:2463] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 1.
I20250110 03:06:52.503886 17391 leader_election.cc:304] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.503981 18437 raft_consensus.cc:2798] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:52.504034 18437 raft_consensus.cc:695] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Leader. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:06:52.504079 18437 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.504590 17312 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129). New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:52.584259 18493 raft_consensus.cc:491] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:52.584347 18493 raft_consensus.cc:513] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.584491 18493 leader_election.cc:290] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:52.587007 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2221475806475c83aa9867a548f51f" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b" is_pre_election: true
I20250110 03:06:52.587100 17457 raft_consensus.cc:2463] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 0.
I20250110 03:06:52.587245 17652 leader_election.cc:304] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.587347 18493 raft_consensus.cc:2798] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:52.587400 18493 raft_consensus.cc:491] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:52.587424 18493 raft_consensus.cc:3054] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.588029 18493 raft_consensus.cc:513] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.588141 18493 leader_election.cc:290] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:52.588290 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b2221475806475c83aa9867a548f51f" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:52.588346 17457 raft_consensus.cc:3054] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.588831 17457 raft_consensus.cc:2463] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 1.
I20250110 03:06:52.588968 17652 leader_election.cc:304] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.589043 18493 raft_consensus.cc:2798] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:52.589099 18493 raft_consensus.cc:695] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Becoming Leader. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:52.589151 18493 consensus_queue.cc:237] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:52.589694 17312 catalog_manager.cc:5526] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:52.922346 18437 raft_consensus.cc:491] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:52.922422 18437 raft_consensus.cc:513] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.922567 18437 leader_election.cc:290] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:52.922562 18346 raft_consensus.cc:491] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:52.922629 18346 raft_consensus.cc:513] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.922751 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b6b5410b4e4405b419e22daf379c6a" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e" is_pre_election: true
I20250110 03:06:52.922749 18346 leader_election.cc:290] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:52.922847 17588 raft_consensus.cc:2463] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 0.
I20250110 03:06:52.922916 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b6b5410b4e4405b419e22daf379c6a" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b" is_pre_election: true
I20250110 03:06:52.922983 17457 raft_consensus.cc:2463] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 0.
I20250110 03:06:52.922981 17391 leader_election.cc:304] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.923123 18437 raft_consensus.cc:2798] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:52.923163 18437 raft_consensus.cc:491] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:52.923153 17521 leader_election.cc:304] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:52.923187 18437 raft_consensus.cc:3054] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.923236 18346 raft_consensus.cc:2798] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:52.923269 18346 raft_consensus.cc:491] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:52.923291 18346 raft_consensus.cc:3054] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:52.923753 18437 raft_consensus.cc:513] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.923842 18437 leader_election.cc:290] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:52.923864 18346 raft_consensus.cc:513] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:52.923966 18346 leader_election.cc:290] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:52.924024 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b6b5410b4e4405b419e22daf379c6a" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:52.924077 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b6b5410b4e4405b419e22daf379c6a" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:52.924103 17588 raft_consensus.cc:2388] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 92bcfebb99cc47bba372938b64a40f4b in current term 1: Already voted for candidate 74fcf6c6434b49d8a777aaeda45d971e in this term.
I20250110 03:06:52.924144 17457 raft_consensus.cc:2388] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 74fcf6c6434b49d8a777aaeda45d971e in current term 1: Already voted for candidate 92bcfebb99cc47bba372938b64a40f4b in this term.
I20250110 03:06:52.924245 17391 leader_election.cc:304] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 92bcfebb99cc47bba372938b64a40f4b; no voters: 74fcf6c6434b49d8a777aaeda45d971e
I20250110 03:06:52.924301 17521 leader_election.cc:304] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e; no voters: 92bcfebb99cc47bba372938b64a40f4b
I20250110 03:06:52.924319 18437 raft_consensus.cc:2743] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 03:06:52.924374 18346 raft_consensus.cc:2743] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 03:06:53.054598 18437 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:53.213477 18493 consensus_queue.cc:1035] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:53.491780 18493 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:53.491868 18493 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:53.491976 18493 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
W20250110 03:06:53.492571 17652 leader_election.cc:336] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693): Network error: Client connection negotiation failed: client connection to 127.14.19.134:36693: connect: Connection refused (error 111)
I20250110 03:06:53.492619 17652 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59; no voters: 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:53.492705 18493 raft_consensus.cc:2743] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 03:06:54.476198 18522 raft_consensus.cc:491] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:54.476285 18522 raft_consensus.cc:513] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:54.476397 18522 leader_election.cc:290] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:54.476603 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b6b5410b4e4405b419e22daf379c6a" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b" is_pre_election: true
I20250110 03:06:54.476688 17457 raft_consensus.cc:2463] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 1.
I20250110 03:06:54.476840 17521 leader_election.cc:304] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:54.476958 18522 raft_consensus.cc:2798] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 03:06:54.477006 18522 raft_consensus.cc:491] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:54.477028 18522 raft_consensus.cc:3054] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:54.477644 18522 raft_consensus.cc:513] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:54.477732 18522 leader_election.cc:290] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:54.477866 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8b6b5410b4e4405b419e22daf379c6a" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:54.477924 17457 raft_consensus.cc:3054] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:54.478408 17457 raft_consensus.cc:2463] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 2.
I20250110 03:06:54.478572 17521 leader_election.cc:304] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:54.478652 18522 raft_consensus.cc:2798] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:54.478767 18522 raft_consensus.cc:695] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:54.478814 18522 consensus_queue.cc:237] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } }
I20250110 03:06:54.479255 17313 catalog_manager.cc:5526] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 0 to 2, leader changed from <none> to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:54.613544 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:54.613723 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:54.614930 18529 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20250110 03:06:54.649941 17457 raft_consensus.cc:1270] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250110 03:06:54.650314 18522 consensus_queue.cc:1035] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 22.7262 ms
time per row: 0.0113631 ms
I20250110 03:06:54.660733 17313 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60734:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:06:54.662184 17699 tablet_service.cc:1467] Processing CreateTablet for tablet 567885520d33460fae980ae73fabcd6a (DEFAULT_TABLE table=rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:54.662191 17435 tablet_service.cc:1467] Processing CreateTablet for tablet 10fb55ba712c42ee95fcf9f300a54ea0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:54.662228 17566 tablet_service.cc:1467] Processing CreateTablet for tablet 567885520d33460fae980ae73fabcd6a (DEFAULT_TABLE table=rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:54.662228 17567 tablet_service.cc:1467] Processing CreateTablet for tablet 10fb55ba712c42ee95fcf9f300a54ea0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:54.662350 17566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567885520d33460fae980ae73fabcd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:54.662351 17699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567885520d33460fae980ae73fabcd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:54.662351 17435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fb55ba712c42ee95fcf9f300a54ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:54.662590 17567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fb55ba712c42ee95fcf9f300a54ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:54.663625 17697 tablet_service.cc:1467] Processing CreateTablet for tablet 85fe921f114943c896a041599c39dbe5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:54.663686 18579 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:54.663736 17697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe921f114943c896a041599c39dbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:54.663789 17568 tablet_service.cc:1467] Processing CreateTablet for tablet 85fe921f114943c896a041599c39dbe5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:06:54.663882 17568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe921f114943c896a041599c39dbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:54.664074 18579 tablet_bootstrap.cc:654] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:54.664434 18578 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:54.664654 18579 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:54.664705 18579 ts_tablet_manager.cc:1397] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:54.664848 18579 raft_consensus.cc:357] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.664916 18580 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b: Bootstrap starting.
I20250110 03:06:54.664930 18579 raft_consensus.cc:383] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:54.664959 18579 raft_consensus.cc:738] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:54.665004 18579 consensus_queue.cc:260] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.665237 18579 ts_tablet_manager.cc:1428] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:54.665303 18579 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: Bootstrap starting.
I20250110 03:06:54.665315 18580 tablet_bootstrap.cc:654] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:54.665820 18579 tablet_bootstrap.cc:654] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:54.665908 18580 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b: No bootstrap required, opened a new log
I20250110 03:06:54.665997 18580 ts_tablet_manager.cc:1397] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:54.666146 18580 raft_consensus.cc:357] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.666195 18580 raft_consensus.cc:383] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:54.666217 18580 raft_consensus.cc:738] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Initialized, Role: FOLLOWER
I20250110 03:06:54.666286 18580 consensus_queue.cc:260] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.666376 18580 ts_tablet_manager.cc:1428] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:54.666467 18578 tablet_bootstrap.cc:654] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:54.666893 18579 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: No bootstrap required, opened a new log
I20250110 03:06:54.666946 18579 ts_tablet_manager.cc:1397] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:54.667032 18578 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:54.667081 18578 ts_tablet_manager.cc:1397] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:06:54.667094 18579 raft_consensus.cc:357] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.667160 18579 raft_consensus.cc:383] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:54.667224 18579 raft_consensus.cc:738] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Initialized, Role: FOLLOWER
I20250110 03:06:54.667227 18578 raft_consensus.cc:357] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.667294 18578 raft_consensus.cc:383] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:54.667300 18579 consensus_queue.cc:260] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.667354 18578 raft_consensus.cc:738] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:54.667421 18578 consensus_queue.cc:260] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.667459 18579 ts_tablet_manager.cc:1428] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:54.667635 18578 ts_tablet_manager.cc:1428] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:54.667737 18578 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:54.668321 18578 tablet_bootstrap.cc:654] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:54.668800 18578 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:54.668838 18578 ts_tablet_manager.cc:1397] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:54.668969 18578 raft_consensus.cc:357] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.669013 18578 raft_consensus.cc:383] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:54.669036 18578 raft_consensus.cc:738] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:54.669088 18578 consensus_queue.cc:260] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.669184 18578 ts_tablet_manager.cc:1428] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:54.669242 18578 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Bootstrap starting.
I20250110 03:06:54.669621 18578 tablet_bootstrap.cc:654] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Neither blocks nor log segments found. Creating new log.
I20250110 03:06:54.670100 18578 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: No bootstrap required, opened a new log
I20250110 03:06:54.670138 18578 ts_tablet_manager.cc:1397] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:54.670258 18578 raft_consensus.cc:357] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.670300 18578 raft_consensus.cc:383] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:06:54.670323 18578 raft_consensus.cc:738] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Initialized, Role: FOLLOWER
I20250110 03:06:54.670392 18578 consensus_queue.cc:260] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.670483 18578 ts_tablet_manager.cc:1428] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:54.725620 18522 raft_consensus.cc:491] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:54.725713 18522 raft_consensus.cc:513] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.725878 18522 leader_election.cc:290] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:54.728358 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fe921f114943c896a041599c39dbe5" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:54.728440 17718 raft_consensus.cc:2463] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 0.
I20250110 03:06:54.728586 17522 leader_election.cc:304] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:54.728709 18522 raft_consensus.cc:2798] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:54.728756 18522 raft_consensus.cc:491] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:54.728771 18522 raft_consensus.cc:3054] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:54.729386 18522 raft_consensus.cc:513] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.729481 18522 leader_election.cc:290] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:54.729643 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fe921f114943c896a041599c39dbe5" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:54.729700 17718 raft_consensus.cc:3054] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:54.730228 17718 raft_consensus.cc:2463] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 1.
I20250110 03:06:54.730368 17522 leader_election.cc:304] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:54.730459 18522 raft_consensus.cc:2798] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:54.730520 18522 raft_consensus.cc:695] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:54.730589 18522 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } }
I20250110 03:06:54.731127 17313 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:54.766983 18522 raft_consensus.cc:491] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:54.767069 18522 raft_consensus.cc:513] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.767189 18522 leader_election.cc:290] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:54.767366 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567885520d33460fae980ae73fabcd6a" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:54.767433 17718 raft_consensus.cc:2463] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 0.
I20250110 03:06:54.767560 17522 leader_election.cc:304] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:54.767665 18522 raft_consensus.cc:2798] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:54.767709 18522 raft_consensus.cc:491] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:54.767736 18522 raft_consensus.cc:3054] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:54.768287 18522 raft_consensus.cc:513] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.768409 18522 leader_election.cc:290] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:54.768577 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567885520d33460fae980ae73fabcd6a" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:54.768643 17718 raft_consensus.cc:3054] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:54.769107 17718 raft_consensus.cc:2463] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 1.
I20250110 03:06:54.769263 17522 leader_election.cc:304] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:54.769346 18522 raft_consensus.cc:2798] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:54.769410 18522 raft_consensus.cc:695] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:54.769460 18522 consensus_queue.cc:237] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.770018 17313 catalog_manager.cc:5526] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:54.835884 18522 raft_consensus.cc:491] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:54.835973 18522 raft_consensus.cc:513] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.836109 18522 leader_election.cc:290] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:54.836333 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b" is_pre_election: true
I20250110 03:06:54.836426 17457 raft_consensus.cc:2463] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 0.
I20250110 03:06:54.836591 17521 leader_election.cc:304] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:54.836687 18522 raft_consensus.cc:2798] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:06:54.836743 18522 raft_consensus.cc:491] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:54.836762 18522 raft_consensus.cc:3054] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:54.837332 18522 raft_consensus.cc:513] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.837421 18522 leader_election.cc:290] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:54.837608 17457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:54.837669 17457 raft_consensus.cc:3054] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:06:54.838205 17457 raft_consensus.cc:2463] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 1.
I20250110 03:06:54.838384 17521 leader_election.cc:304] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:06:54.838466 18522 raft_consensus.cc:2798] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:06:54.838512 18522 raft_consensus.cc:695] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:54.838584 18522 consensus_queue.cc:237] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } }
I20250110 03:06:54.839202 17313 catalog_manager.cc:5526] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 0 to 1, leader changed from <none> to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } }
W20250110 03:06:54.916849 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:54.917060 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:54.918422 18591 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20250110 03:06:54.947608 17457 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:54.947850 17718 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:54.948261 18618 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:54.948400 17718 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:06:54.948443 18618 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:06:54.949133 18523 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 19.5361 ms
time per row: 0.00976807 ms
I20250110 03:06:54.960922 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:38659
--local_ip_for_outbound_sockets=127.14.19.132
--tserver_master_addrs=127.14.19.190:35275
--webserver_port=35305
--webserver_interface=127.14.19.132
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:06:55.033859 18624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:55.034020 18624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:55.034053 18624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:55.035343 18624 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:55.035384 18624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:55.035424 18624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:06:55.036870 18624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:38659
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=35305
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.18624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:55.037114 18624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:55.037317 18624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:55.042042 18632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.042857 18630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.043658 18629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:55.044723 18624 server_base.cc:1034] running on GCE node
I20250110 03:06:55.044916 18624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:55.045084 18624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:55.046206 18624 hybrid_clock.cc:648] HybridClock initialized: now 1736478415046187 us; error 35 us; skew 500 ppm
I20250110 03:06:55.047389 18624 webserver.cc:458] Webserver started at http://127.14.19.132:35305/ using document root <none> and password file <none>
I20250110 03:06:55.047574 18624 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:55.047650 18624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:55.048779 18624 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250110 03:06:55.049253 18638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.049396 18624 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.049472 18624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b61b88548de84191b17de06cca3c435a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:55.049710 18624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:55.066538 18624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:55.066769 18624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:55.066864 18624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:55.067052 18624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:55.067323 18624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:55.067355 18624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.067386 18624 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:55.067415 18624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.072747 18624 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:38659
I20250110 03:06:55.072795 18751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:38659 every 8 connection(s)
I20250110 03:06:55.073076 18624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:06:55.074735 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 18624
I20250110 03:06:55.075006 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:39385
--local_ip_for_outbound_sockets=127.14.19.133
--tserver_master_addrs=127.14.19.190:35275
--webserver_port=39437
--webserver_interface=127.14.19.133
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:55.078189 18752 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:55.078281 18752 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:55.078449 18752 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:55.078812 17313 ts_manager.cc:194] Re-registered known tserver with Master: b61b88548de84191b17de06cca3c435a (127.14.19.132:38659)
I20250110 03:06:55.079195 17313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:39357
W20250110 03:06:55.148345 18755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:55.148499 18755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:55.148527 18755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:55.149699 18755 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:55.149739 18755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:55.149775 18755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:06:55.151191 18755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:39385
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=39437
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.18755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:55.151396 18755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:55.151587 18755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:55.156147 18763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.156767 18761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.157752 18760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:55.158605 18755 server_base.cc:1034] running on GCE node
I20250110 03:06:55.158779 18755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:55.158987 18755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:55.160115 18755 hybrid_clock.cc:648] HybridClock initialized: now 1736478415160097 us; error 31 us; skew 500 ppm
I20250110 03:06:55.166047 18755 webserver.cc:458] Webserver started at http://127.14.19.133:39437/ using document root <none> and password file <none>
I20250110 03:06:55.166235 18755 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:55.166301 18755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:55.167543 18755 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:55.168030 18769 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.168160 18755 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.168229 18755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1e6d3913b04475f8962dd6496fbea0a"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:55.168486 18755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:55.179190 18755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:55.179394 18755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:55.179497 18755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:55.179648 18755 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:55.179908 18755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:55.179948 18755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.179980 18755 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:55.180009 18755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.185168 18755 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:39385
I20250110 03:06:55.185221 18882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:39385 every 8 connection(s)
I20250110 03:06:55.185516 18755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:06:55.189422 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 18755
I20250110 03:06:55.189632 18883 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:55.189657 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:36693
--local_ip_for_outbound_sockets=127.14.19.134
--tserver_master_addrs=127.14.19.190:35275
--webserver_port=42573
--webserver_interface=127.14.19.134
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:55.189711 18883 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:55.189879 18883 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:55.190228 17313 ts_manager.cc:194] Re-registered known tserver with Master: d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385)
I20250110 03:06:55.190704 17313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:43373
W20250110 03:06:55.267338 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:55.267545 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:55.267578 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:55.269452 18886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:55.269510 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:55.269551 18886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:06:55.271728 18886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:36693
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=42573
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:55.271929 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:55.272194 18886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:55.277263 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.277978 18892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.279058 18891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:55.279832 18886 server_base.cc:1034] running on GCE node
I20250110 03:06:55.279990 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:55.280167 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:55.281299 18886 hybrid_clock.cc:648] HybridClock initialized: now 1736478415281286 us; error 36 us; skew 500 ppm
I20250110 03:06:55.282353 18886 webserver.cc:458] Webserver started at http://127.14.19.134:42573/ using document root <none> and password file <none>
I20250110 03:06:55.282518 18886 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:55.282593 18886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:55.283679 18886 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:55.284240 18900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.284415 18886 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:55.284480 18886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4ce3080cc7de4c729423b95fb7158a07"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:55.284734 18886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:55.291826 18886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:55.292006 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:55.292098 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:55.292248 18886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:55.292610 18907 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250110 03:06:55.293280 18886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250110 03:06:55.293321 18886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:55.293365 18886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250110 03:06:55.293941 18886 ts_tablet_manager.cc:610] Registered 1 tablets
I20250110 03:06:55.293972 18886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:55.294003 18907 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap starting.
I20250110 03:06:55.299654 18886 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:36693
I20250110 03:06:55.299728 19014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:36693 every 8 connection(s)
I20250110 03:06:55.299974 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:06:55.302809 18907 log.cc:826] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:55.303704 18907 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:55.303860 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 18886
I20250110 03:06:55.304008 18907 tablet_bootstrap.cc:492] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap complete.
I20250110 03:06:55.304152 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.135:32779
--local_ip_for_outbound_sockets=127.14.19.135
--tserver_master_addrs=127.14.19.190:35275
--webserver_port=33191
--webserver_interface=127.14.19.135
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:06:55.304790 18907 ts_tablet_manager.cc:1397] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250110 03:06:55.306432 18907 raft_consensus.cc:357] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:55.306753 18907 raft_consensus.cc:738] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Initialized, Role: FOLLOWER
I20250110 03:06:55.306878 18907 consensus_queue.cc:260] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:55.307137 18907 ts_tablet_manager.cc:1428] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250110 03:06:55.307251 19015 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:55.307346 19015 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:55.307544 19015 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:55.308104 17313 ts_manager.cc:194] Re-registered known tserver with Master: 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
I20250110 03:06:55.308508 17313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:44267
I20250110 03:06:55.309079 19015 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:06:55.372735 18569 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:55.372822 18569 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:55.372977 18569 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
I20250110 03:06:55.376374 18969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888f2d12f3254197a6a490e23016bd46" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce3080cc7de4c729423b95fb7158a07" is_pre_election: true
I20250110 03:06:55.376518 18969 raft_consensus.cc:2405] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b345888171f4abcb611a51e27cbcd59 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250110 03:06:55.376794 17652 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59; no voters: 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:55.376906 18569 raft_consensus.cc:2743] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 03:06:55.380312 19019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:06:55.380442 19019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:06:55.380460 19019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:06:55.381624 19019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:06:55.381657 19019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:06:55.381680 19019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.135
I20250110 03:06:55.383020 19019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.135:32779
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.19.135
--webserver_port=33191
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.19019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.135
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:06:55.383183 19019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:06:55.383345 19019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:06:55.387804 19027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.388775 19028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:06:55.389587 19030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:06:55.390370 19019 server_base.cc:1034] running on GCE node
I20250110 03:06:55.390527 19019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:06:55.390753 19019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:06:55.391876 19019 hybrid_clock.cc:648] HybridClock initialized: now 1736478415391858 us; error 28 us; skew 500 ppm
I20250110 03:06:55.392910 19019 webserver.cc:458] Webserver started at http://127.14.19.135:33191/ using document root <none> and password file <none>
I20250110 03:06:55.393090 19019 fs_manager.cc:362] Metadata directory not provided
I20250110 03:06:55.393150 19019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:06:55.394207 19019 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:06:55.394703 19036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.394846 19019 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.394913 19019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d28dc7130e374d8486cc68018c03fd34"
format_stamp: "Formatted at 2025-01-10 03:06:46 on dist-test-slave-nrvn"
I20250110 03:06:55.395146 19019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:06:55.406421 19019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:06:55.406652 19019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:06:55.406756 19019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:06:55.406912 19019 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:06:55.407177 19019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:06:55.407210 19019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.407238 19019 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:06:55.407270 19019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:55.412442 19019 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.135:32779
I20250110 03:06:55.412484 19149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.135:32779 every 8 connection(s)
I20250110 03:06:55.412781 19019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250110 03:06:55.417044 19150 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:06:55.417133 19150 heartbeater.cc:463] Registering TS with master...
I20250110 03:06:55.417289 19150 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:06:55.417582 17313 ts_manager.cc:194] Re-registered known tserver with Master: d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779)
I20250110 03:06:55.417906 17313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.135:35377
I20250110 03:06:55.419260 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 19019
I20250110 03:06:55.419373 14414 test_util.cc:274] Using random seed: 1662810092
I20250110 03:06:55.424132 14414 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20250110 03:06:55.424250 14414 test_util.cc:274] Using random seed: 1662814969
I20250110 03:06:55.429603 14414 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20250110 03:06:55.429713 14414 test_util.cc:274] Using random seed: 1662820433
I20250110 03:06:55.435113 14414 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20250110 03:06:55.435218 14414 test_util.cc:274] Using random seed: 1662825937
I20250110 03:06:55.444159 14414 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20250110 03:06:55.444260 14414 test_util.cc:274] Using random seed: 1662834980
I20250110 03:06:55.453698 14414 test_workload.cc:405] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20250110 03:06:55.550424 19016 tablet.cc:2367] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:06:55.657984 17568 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:55.660106 18817 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.663556 17697 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:06:55.675320 19084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.683602 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.710155 17435 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:55.710254 18949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.831380 17435 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:55.832517 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.832734 17697 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:06:55.833740 19084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.835402 18817 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.847564 18949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:55.848722 17568 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:06:55.874042 17718 consensus_queue.cc:237] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:55.874708 17588 raft_consensus.cc:1270] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20250110 03:06:55.874912 19223 consensus_queue.cc:1035] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s
I20250110 03:06:55.875806 19223 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:55.877126 17588 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:55.877188 17309 catalog_manager.cc:5526] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index -1 to 71, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:55.879084 17654 consensus_peers.cc:487] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 -> Peer d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385): Couldn't send request to peer d1e6d3913b04475f8962dd6496fbea0a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8dec030e45a84eb89321f275d0013826. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:55.893225 17719 consensus_queue.cc:237] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:55.896324 17456 raft_consensus.cc:1270] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20250110 03:06:55.896517 19215 consensus_queue.cc:1035] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20250110 03:06:55.897126 19215 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:55.897511 17457 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
W20250110 03:06:55.897909 17655 consensus_peers.cc:487] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 -> Peer d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779): Couldn't send request to peer d28dc7130e374d8486cc68018c03fd34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b2221475806475c83aa9867a548f51f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:55.899150 17314 catalog_manager.cc:5526] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index -1 to 66, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:55.934149 17457 consensus_queue.cc:237] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 60, Committed index: 60, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:55.935703 17719 raft_consensus.cc:1270] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch)
I20250110 03:06:55.937942 19212 consensus_queue.cc:1035] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 60, Time since last communication: 0.000s
I20250110 03:06:55.939316 18437 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:55.940025 17718 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:55.941545 17314 catalog_manager.cc:5526] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index -1 to 61, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
W20250110 03:06:55.945016 17391 consensus_peers.cc:487] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b -> Peer b61b88548de84191b17de06cca3c435a (127.14.19.132:38659): Couldn't send request to peer b61b88548de84191b17de06cca3c435a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8b5177de1fa4e1c9668453452f2064f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:55.999248 17588 consensus_queue.cc:237] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 68, Committed index: 68, Last appended: 1.68, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:55.999701 17718 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 68. Preceding OpId from leader: term: 1 index: 69. (index mismatch)
I20250110 03:06:56.000057 19222 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 69, Last known committed idx: 68, Time since last communication: 0.000s
I20250110 03:06:56.002619 19222 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } } }
I20250110 03:06:56.002878 17718 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.69: config changed from index -1 to 69, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New config: { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } } }
I20250110 03:06:56.003947 17314 catalog_manager.cc:5526] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index -1 to 69, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:06:56.005111 17521 consensus_peers.cc:487] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e -> Peer 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693): Couldn't send request to peer 4ce3080cc7de4c729423b95fb7158a07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 567885520d33460fae980ae73fabcd6a. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.016746 17588 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.017330 17457 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20250110 03:06:56.017844 19222 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 90, Time since last communication: 0.000s
I20250110 03:06:56.019460 19222 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:56.020762 17309 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index -1 to 92, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.020850 17455 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
W20250110 03:06:56.021694 17523 consensus_peers.cc:487] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e -> Peer d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385): Couldn't send request to peer d1e6d3913b04475f8962dd6496fbea0a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7906c51301244bd4a592d8e01e6ad602. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.034476 17717 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
W20250110 03:06:56.035312 17655 consensus_peers.cc:487] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 -> Peer d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779): Couldn't send request to peer d28dc7130e374d8486cc68018c03fd34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4d68bc455bd4920ba99280f593678ec. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.035758 17586 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20250110 03:06:56.036641 19182 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20250110 03:06:56.038010 19182 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:56.039379 17309 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index -1 to 79, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.039758 17586 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:56.050921 17455 consensus_queue.cc:237] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.051945 17587 raft_consensus.cc:1270] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
W20250110 03:06:56.053150 17391 consensus_peers.cc:487] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b -> Peer b61b88548de84191b17de06cca3c435a (127.14.19.132:38659): Couldn't send request to peer b61b88548de84191b17de06cca3c435a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83d2ceb09bc5442f8c329ca6b3ad6f35. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.053417 19171 consensus_queue.cc:1035] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20250110 03:06:56.054792 19212 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.056298 17314 catalog_manager.cc:5526] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index -1 to 89, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.056401 17586 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.065485 17586 consensus_queue.cc:237] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:56.066073 17456 raft_consensus.cc:1270] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
W20250110 03:06:56.066249 17521 consensus_peers.cc:487] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e -> Peer 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693): Couldn't send request to peer 4ce3080cc7de4c729423b95fb7158a07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa58142d4f86446b99561d0a0fb9fc96. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.066282 19222 consensus_queue.cc:1035] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s
I20250110 03:06:56.067651 18522 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } } }
I20250110 03:06:56.068797 17314 catalog_manager.cc:5526] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index -1 to 108, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.070158 17455 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } } }
I20250110 03:06:56.079888 18752 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:06:56.081354 17588 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
W20250110 03:06:56.082269 17523 consensus_peers.cc:487] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e -> Peer d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385): Couldn't send request to peer d1e6d3913b04475f8962dd6496fbea0a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85fe921f114943c896a041599c39dbe5. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.082541 17719 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20250110 03:06:56.083073 18618 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20250110 03:06:56.083990 19222 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:56.084115 17719 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:56.085321 17314 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index -1 to 96, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:56.096750 17457 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
W20250110 03:06:56.097486 17391 consensus_peers.cc:487] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b -> Peer b61b88548de84191b17de06cca3c435a (127.14.19.132:38659): Couldn't send request to peer b61b88548de84191b17de06cca3c435a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 593fb2f6a0a7417ca54b800249318eae. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.097784 17717 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20250110 03:06:56.098325 19318 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20250110 03:06:56.101814 19318 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.104677 17309 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index -1 to 82, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.105720 17719 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.116288 17457 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.116719 17718 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20250110 03:06:56.116982 19318 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20250110 03:06:56.118189 19212 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
W20250110 03:06:56.119897 17393 consensus_peers.cc:487] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b -> Peer d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779): Couldn't send request to peer d28dc7130e374d8486cc68018c03fd34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e365416b475a4288b17d235deddf139d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.121527 17309 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index -1 to 80, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.122421 17719 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:56.129459 17588 consensus_queue.cc:237] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 2.61, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.130055 17456 raft_consensus.cc:1270] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 61. Preceding OpId from leader: term: 2 index: 62. (index mismatch)
W20250110 03:06:56.130148 17523 consensus_peers.cc:487] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e -> Peer d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385): Couldn't send request to peer d1e6d3913b04475f8962dd6496fbea0a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8b6b5410b4e4405b419e22daf379c6a. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.130345 19222 consensus_queue.cc:1035] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s
I20250110 03:06:56.131079 18618 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.62: config changed from index -1 to 62, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:56.132098 17456 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Committing config change with OpId 2.62: config changed from index -1 to 62, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } } }
I20250110 03:06:56.132699 17309 catalog_manager.cc:5526] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index -1 to 62, NON_VOTER d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) added. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.145148 17457 consensus_queue.cc:237] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.145907 17585 raft_consensus.cc:1270] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
W20250110 03:06:56.146137 17391 consensus_peers.cc:487] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b -> Peer b61b88548de84191b17de06cca3c435a (127.14.19.132:38659): Couldn't send request to peer b61b88548de84191b17de06cca3c435a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3ec6ddcb58d48d89f4cea0b3db80035. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:56.146186 19318 consensus_queue.cc:1035] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
I20250110 03:06:56.147508 19318 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index -1 to 91, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.147608 17585 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index -1 to 91, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.148583 17314 catalog_manager.cc:5526] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index -1 to 91, NON_VOTER b61b88548de84191b17de06cca3c435a (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } } }
I20250110 03:06:56.191375 18883 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:06:56.318919 19444 ts_tablet_manager.cc:927] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Initiating tablet copy from peer 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:56.319129 19444 tablet_copy_client.cc:323] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:46605
I20250110 03:06:56.323530 17739 tablet_copy_service.cc:140] P 0b345888171f4abcb611a51e27cbcd59: Received BeginTabletCopySession request for tablet 8dec030e45a84eb89321f275d0013826 from peer d1e6d3913b04475f8962dd6496fbea0a ({username='slave'} at 127.14.19.133:58677)
I20250110 03:06:56.323611 17739 tablet_copy_service.cc:161] P 0b345888171f4abcb611a51e27cbcd59: Beginning new tablet copy session on tablet 8dec030e45a84eb89321f275d0013826 from peer d1e6d3913b04475f8962dd6496fbea0a at {username='slave'} at 127.14.19.133:58677: session id = d1e6d3913b04475f8962dd6496fbea0a-8dec030e45a84eb89321f275d0013826
I20250110 03:06:56.324221 17739 tablet_copy_source_session.cc:215] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.325098 19444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dec030e45a84eb89321f275d0013826. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.328172 19444 tablet_copy_client.cc:806] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.328330 19444 tablet_copy_client.cc:670] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.329999 19444 tablet_copy_client.cc:538] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.330829 19444 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap starting.
I20250110 03:06:56.335429 19444 log.cc:826] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:56.351882 19444 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=145 overwritten=0 applied=144 ignored=0} inserts{seen=829 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:56.352196 19444 tablet_bootstrap.cc:492] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap complete.
I20250110 03:06:56.352342 19444 ts_tablet_manager.cc:1397] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.013s
I20250110 03:06:56.353652 19444 raft_consensus.cc:357] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.354236 19444 raft_consensus.cc:738] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1e6d3913b04475f8962dd6496fbea0a, State: Initialized, Role: LEARNER
I20250110 03:06:56.354326 19444 consensus_queue.cc:260] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 144, Last appended: 1.145, Last appended by leader: 145, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.354594 19444 ts_tablet_manager.cc:1428] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20250110 03:06:56.355203 17739 tablet_copy_service.cc:342] P 0b345888171f4abcb611a51e27cbcd59: Request end of tablet copy session d1e6d3913b04475f8962dd6496fbea0a-8dec030e45a84eb89321f275d0013826 received from {username='slave'} at 127.14.19.133:58677
I20250110 03:06:56.355269 17739 tablet_copy_service.cc:434] P 0b345888171f4abcb611a51e27cbcd59: ending tablet copy session d1e6d3913b04475f8962dd6496fbea0a-8dec030e45a84eb89321f275d0013826 on tablet 8dec030e45a84eb89321f275d0013826 with peer d1e6d3913b04475f8962dd6496fbea0a
I20250110 03:06:56.380463 19451 ts_tablet_manager.cc:927] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Initiating tablet copy from peer 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:56.381644 19451 tablet_copy_client.cc:323] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:46605
I20250110 03:06:56.387288 17739 tablet_copy_service.cc:140] P 0b345888171f4abcb611a51e27cbcd59: Received BeginTabletCopySession request for tablet 8b2221475806475c83aa9867a548f51f from peer d28dc7130e374d8486cc68018c03fd34 ({username='slave'} at 127.14.19.135:35227)
I20250110 03:06:56.387351 17739 tablet_copy_service.cc:161] P 0b345888171f4abcb611a51e27cbcd59: Beginning new tablet copy session on tablet 8b2221475806475c83aa9867a548f51f from peer d28dc7130e374d8486cc68018c03fd34 at {username='slave'} at 127.14.19.135:35227: session id = d28dc7130e374d8486cc68018c03fd34-8b2221475806475c83aa9867a548f51f
I20250110 03:06:56.387943 17739 tablet_copy_source_session.cc:215] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.388906 19451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2221475806475c83aa9867a548f51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.395052 19451 tablet_copy_client.cc:806] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.395349 19451 tablet_copy_client.cc:670] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.397409 19451 tablet_copy_client.cc:538] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.397827 19454 ts_tablet_manager.cc:927] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Initiating tablet copy from peer 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:56.398156 19454 tablet_copy_client.cc:323] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42521
I20250110 03:06:56.398403 19451 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Bootstrap starting.
I20250110 03:06:56.403306 19457 ts_tablet_manager.cc:927] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:56.403702 19457 tablet_copy_client.cc:323] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:06:56.403746 19451 log.cc:826] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:56.413244 17477 tablet_copy_service.cc:140] P 92bcfebb99cc47bba372938b64a40f4b: Received BeginTabletCopySession request for tablet f8b5177de1fa4e1c9668453452f2064f from peer b61b88548de84191b17de06cca3c435a ({username='slave'} at 127.14.19.132:35497)
I20250110 03:06:56.413327 17477 tablet_copy_service.cc:161] P 92bcfebb99cc47bba372938b64a40f4b: Beginning new tablet copy session on tablet f8b5177de1fa4e1c9668453452f2064f from peer b61b88548de84191b17de06cca3c435a at {username='slave'} at 127.14.19.132:35497: session id = b61b88548de84191b17de06cca3c435a-f8b5177de1fa4e1c9668453452f2064f
I20250110 03:06:56.414016 17477 tablet_copy_source_session.cc:215] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.415159 19454 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8b5177de1fa4e1c9668453452f2064f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.418025 19454 tablet_copy_client.cc:806] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.418174 19454 tablet_copy_client.cc:670] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.418526 19150 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:06:56.422689 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet 567885520d33460fae980ae73fabcd6a from peer 4ce3080cc7de4c729423b95fb7158a07 ({username='slave'} at 127.14.19.134:39939)
I20250110 03:06:56.422762 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet 567885520d33460fae980ae73fabcd6a from peer 4ce3080cc7de4c729423b95fb7158a07 at {username='slave'} at 127.14.19.134:39939: session id = 4ce3080cc7de4c729423b95fb7158a07-567885520d33460fae980ae73fabcd6a
I20250110 03:06:56.423461 19454 tablet_copy_client.cc:538] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.423784 17608 tablet_copy_source_session.cc:215] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.424360 19457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567885520d33460fae980ae73fabcd6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.424703 19454 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Bootstrap starting.
I20250110 03:06:56.426263 19457 tablet_copy_client.cc:806] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.426374 19457 tablet_copy_client.cc:670] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.427340 19451 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=123 overwritten=0 applied=123 ignored=0} inserts{seen=808 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.427558 19451 tablet_bootstrap.cc:492] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Bootstrap complete.
I20250110 03:06:56.427699 19451 ts_tablet_manager.cc:1397] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.003s
I20250110 03:06:56.428478 19451 raft_consensus.cc:357] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.428597 19451 raft_consensus.cc:738] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d28dc7130e374d8486cc68018c03fd34, State: Initialized, Role: LEARNER
I20250110 03:06:56.428669 19454 log.cc:826] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Log is configured to *not* fsync() on all Append() calls
I20250110 03:06:56.428668 19451 consensus_queue.cc:260] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 123, Last appended: 1.123, Last appended by leader: 123, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.428917 19451 ts_tablet_manager.cc:1428] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250110 03:06:56.429425 17739 tablet_copy_service.cc:342] P 0b345888171f4abcb611a51e27cbcd59: Request end of tablet copy session d28dc7130e374d8486cc68018c03fd34-8b2221475806475c83aa9867a548f51f received from {username='slave'} at 127.14.19.135:35227
I20250110 03:06:56.429484 17739 tablet_copy_service.cc:434] P 0b345888171f4abcb611a51e27cbcd59: ending tablet copy session d28dc7130e374d8486cc68018c03fd34-8b2221475806475c83aa9867a548f51f on tablet 8b2221475806475c83aa9867a548f51f with peer d28dc7130e374d8486cc68018c03fd34
I20250110 03:06:56.439823 19457 tablet_copy_client.cc:538] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.440622 19457 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap starting.
I20250110 03:06:56.474468 19457 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=122 overwritten=0 applied=121 ignored=0} inserts{seen=806 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:56.474764 19457 tablet_bootstrap.cc:492] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap complete.
I20250110 03:06:56.474890 19457 ts_tablet_manager.cc:1397] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.010s
I20250110 03:06:56.475035 19457 raft_consensus.cc:357] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:56.475673 19457 raft_consensus.cc:738] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Initialized, Role: LEARNER
I20250110 03:06:56.475746 19457 consensus_queue.cc:260] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 121, Last appended: 1.122, Last appended by leader: 122, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 69 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:56.475910 19457 ts_tablet_manager.cc:1428] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:56.477252 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session 4ce3080cc7de4c729423b95fb7158a07-567885520d33460fae980ae73fabcd6a received from {username='slave'} at 127.14.19.134:39939
I20250110 03:06:56.477334 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session 4ce3080cc7de4c729423b95fb7158a07-567885520d33460fae980ae73fabcd6a on tablet 567885520d33460fae980ae73fabcd6a with peer 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:56.483680 19471 ts_tablet_manager.cc:927] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: Initiating tablet copy from peer 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:56.483837 19471 tablet_copy_client.cc:323] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42521
I20250110 03:06:56.484053 19454 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Bootstrap replayed 1/1 log segments. Stats: ops{read=123 overwritten=0 applied=123 ignored=0} inserts{seen=795 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.484059 17477 tablet_copy_service.cc:140] P 92bcfebb99cc47bba372938b64a40f4b: Received BeginTabletCopySession request for tablet 593fb2f6a0a7417ca54b800249318eae from peer b61b88548de84191b17de06cca3c435a ({username='slave'} at 127.14.19.132:35497)
I20250110 03:06:56.484107 17477 tablet_copy_service.cc:161] P 92bcfebb99cc47bba372938b64a40f4b: Beginning new tablet copy session on tablet 593fb2f6a0a7417ca54b800249318eae from peer b61b88548de84191b17de06cca3c435a at {username='slave'} at 127.14.19.132:35497: session id = b61b88548de84191b17de06cca3c435a-593fb2f6a0a7417ca54b800249318eae
I20250110 03:06:56.484398 19454 tablet_bootstrap.cc:492] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Bootstrap complete.
I20250110 03:06:56.484651 17477 tablet_copy_source_session.cc:215] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.484575 19454 ts_tablet_manager.cc:1397] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.008s
I20250110 03:06:56.485539 19471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593fb2f6a0a7417ca54b800249318eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.488505 19454 raft_consensus.cc:357] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.488669 19454 raft_consensus.cc:738] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b61b88548de84191b17de06cca3c435a, State: Initialized, Role: LEARNER
I20250110 03:06:56.488754 19454 consensus_queue.cc:260] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 123, Last appended: 1.123, Last appended by leader: 123, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.489114 19454 ts_tablet_manager.cc:1428] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 03:06:56.489362 17477 tablet_copy_service.cc:342] P 92bcfebb99cc47bba372938b64a40f4b: Request end of tablet copy session b61b88548de84191b17de06cca3c435a-f8b5177de1fa4e1c9668453452f2064f received from {username='slave'} at 127.14.19.132:35497
I20250110 03:06:56.489419 17477 tablet_copy_service.cc:434] P 92bcfebb99cc47bba372938b64a40f4b: ending tablet copy session b61b88548de84191b17de06cca3c435a-f8b5177de1fa4e1c9668453452f2064f on tablet f8b5177de1fa4e1c9668453452f2064f with peer b61b88548de84191b17de06cca3c435a
I20250110 03:06:56.490507 19471 tablet_copy_client.cc:806] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.490646 19471 tablet_copy_client.cc:670] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.494340 19471 tablet_copy_client.cc:538] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.495110 19471 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: Bootstrap starting.
I20250110 03:06:56.497157 19451 ts_tablet_manager.cc:927] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Initiating tablet copy from peer 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:56.497303 19451 tablet_copy_client.cc:323] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:46605
I20250110 03:06:56.497494 17739 tablet_copy_service.cc:140] P 0b345888171f4abcb611a51e27cbcd59: Received BeginTabletCopySession request for tablet c4d68bc455bd4920ba99280f593678ec from peer d28dc7130e374d8486cc68018c03fd34 ({username='slave'} at 127.14.19.135:35227)
I20250110 03:06:56.497560 17739 tablet_copy_service.cc:161] P 0b345888171f4abcb611a51e27cbcd59: Beginning new tablet copy session on tablet c4d68bc455bd4920ba99280f593678ec from peer d28dc7130e374d8486cc68018c03fd34 at {username='slave'} at 127.14.19.135:35227: session id = d28dc7130e374d8486cc68018c03fd34-c4d68bc455bd4920ba99280f593678ec
I20250110 03:06:56.498143 17739 tablet_copy_source_session.cc:215] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.498469 19451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d68bc455bd4920ba99280f593678ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.499791 19451 tablet_copy_client.cc:806] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.499895 19451 tablet_copy_client.cc:670] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.507728 19451 tablet_copy_client.cc:538] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.508493 19451 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Bootstrap starting.
I20250110 03:06:56.510659 19444 ts_tablet_manager.cc:927] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:56.510803 19444 tablet_copy_client.cc:323] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:06:56.514575 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet 7906c51301244bd4a592d8e01e6ad602 from peer d1e6d3913b04475f8962dd6496fbea0a ({username='slave'} at 127.14.19.133:40683)
I20250110 03:06:56.514634 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet 7906c51301244bd4a592d8e01e6ad602 from peer d1e6d3913b04475f8962dd6496fbea0a at {username='slave'} at 127.14.19.133:40683: session id = d1e6d3913b04475f8962dd6496fbea0a-7906c51301244bd4a592d8e01e6ad602
I20250110 03:06:56.515120 17608 tablet_copy_source_session.cc:215] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.515398 19444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7906c51301244bd4a592d8e01e6ad602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.516696 19444 tablet_copy_client.cc:806] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.516805 19444 tablet_copy_client.cc:670] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.519167 19444 tablet_copy_client.cc:538] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.519798 19444 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap starting.
I20250110 03:06:56.526643 19471 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: Bootstrap replayed 1/1 log segments. Stats: ops{read=155 overwritten=0 applied=155 ignored=0} inserts{seen=786 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.526897 19471 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: Bootstrap complete.
I20250110 03:06:56.527006 19471 ts_tablet_manager.cc:1397] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.011s
I20250110 03:06:56.527132 19471 raft_consensus.cc:357] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.527240 19471 raft_consensus.cc:738] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b61b88548de84191b17de06cca3c435a, State: Initialized, Role: LEARNER
I20250110 03:06:56.527273 19471 consensus_queue.cc:260] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.527369 19471 ts_tablet_manager.cc:1428] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:56.527740 17477 tablet_copy_service.cc:342] P 92bcfebb99cc47bba372938b64a40f4b: Request end of tablet copy session b61b88548de84191b17de06cca3c435a-593fb2f6a0a7417ca54b800249318eae received from {username='slave'} at 127.14.19.132:35497
I20250110 03:06:56.527798 17477 tablet_copy_service.cc:434] P 92bcfebb99cc47bba372938b64a40f4b: ending tablet copy session b61b88548de84191b17de06cca3c435a-593fb2f6a0a7417ca54b800249318eae on tablet 593fb2f6a0a7417ca54b800249318eae with peer b61b88548de84191b17de06cca3c435a
I20250110 03:06:56.538421 19505 ts_tablet_manager.cc:927] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:56.538616 19505 tablet_copy_client.cc:323] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:06:56.538810 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet 85fe921f114943c896a041599c39dbe5 from peer d1e6d3913b04475f8962dd6496fbea0a ({username='slave'} at 127.14.19.133:40683)
I20250110 03:06:56.538868 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet 85fe921f114943c896a041599c39dbe5 from peer d1e6d3913b04475f8962dd6496fbea0a at {username='slave'} at 127.14.19.133:40683: session id = d1e6d3913b04475f8962dd6496fbea0a-85fe921f114943c896a041599c39dbe5
I20250110 03:06:56.539448 17608 tablet_copy_source_session.cc:215] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.539933 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe921f114943c896a041599c39dbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.541486 19444 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=112 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.541785 19444 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap complete.
I20250110 03:06:56.541898 19444 ts_tablet_manager.cc:1397] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.003s
I20250110 03:06:56.542011 19444 raft_consensus.cc:357] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.542089 19444 raft_consensus.cc:738] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1e6d3913b04475f8962dd6496fbea0a, State: Initialized, Role: LEARNER
I20250110 03:06:56.542124 19444 consensus_queue.cc:260] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 112, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.542762 19444 ts_tablet_manager.cc:1428] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:56.543370 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session d1e6d3913b04475f8962dd6496fbea0a-7906c51301244bd4a592d8e01e6ad602 received from {username='slave'} at 127.14.19.133:40683
I20250110 03:06:56.543454 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session d1e6d3913b04475f8962dd6496fbea0a-7906c51301244bd4a592d8e01e6ad602 on tablet 7906c51301244bd4a592d8e01e6ad602 with peer d1e6d3913b04475f8962dd6496fbea0a
I20250110 03:06:56.552735 19505 tablet_copy_client.cc:806] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.552873 19505 tablet_copy_client.cc:670] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.560137 19451 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=162 overwritten=0 applied=162 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.560343 19451 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Bootstrap complete.
I20250110 03:06:56.560688 19451 ts_tablet_manager.cc:1397] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.007s
I20250110 03:06:56.560895 19451 raft_consensus.cc:357] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.561036 19451 raft_consensus.cc:738] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d28dc7130e374d8486cc68018c03fd34, State: Initialized, Role: LEARNER
I20250110 03:06:56.561113 19451 consensus_queue.cc:260] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 162, Last appended: 1.162, Last appended by leader: 162, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.561569 19505 tablet_copy_client.cc:538] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.562249 19505 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap starting.
I20250110 03:06:56.566269 19451 ts_tablet_manager.cc:1428] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20250110 03:06:56.577442 17739 tablet_copy_service.cc:342] P 0b345888171f4abcb611a51e27cbcd59: Request end of tablet copy session d28dc7130e374d8486cc68018c03fd34-c4d68bc455bd4920ba99280f593678ec received from {username='slave'} at 127.14.19.135:35227
I20250110 03:06:56.577507 17739 tablet_copy_service.cc:434] P 0b345888171f4abcb611a51e27cbcd59: ending tablet copy session d28dc7130e374d8486cc68018c03fd34-c4d68bc455bd4920ba99280f593678ec on tablet c4d68bc455bd4920ba99280f593678ec with peer d28dc7130e374d8486cc68018c03fd34
I20250110 03:06:56.588716 19505 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=172 overwritten=0 applied=172 ignored=0} inserts{seen=803 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.588995 19505 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap complete.
I20250110 03:06:56.589112 19505 ts_tablet_manager.cc:1397] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.011s
I20250110 03:06:56.589257 19505 raft_consensus.cc:357] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.589323 19505 raft_consensus.cc:738] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1e6d3913b04475f8962dd6496fbea0a, State: Initialized, Role: LEARNER
I20250110 03:06:56.589355 19505 consensus_queue.cc:260] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 172, Last appended: 1.172, Last appended by leader: 172, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.589458 19505 ts_tablet_manager.cc:1428] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:56.589663 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session d1e6d3913b04475f8962dd6496fbea0a-85fe921f114943c896a041599c39dbe5 received from {username='slave'} at 127.14.19.133:40683
I20250110 03:06:56.589733 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session d1e6d3913b04475f8962dd6496fbea0a-85fe921f114943c896a041599c39dbe5 on tablet 85fe921f114943c896a041599c39dbe5 with peer d1e6d3913b04475f8962dd6496fbea0a
I20250110 03:06:56.635744 19471 ts_tablet_manager.cc:927] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: Initiating tablet copy from peer 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:56.635903 19471 tablet_copy_client.cc:323] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42521
I20250110 03:06:56.636826 17477 tablet_copy_service.cc:140] P 92bcfebb99cc47bba372938b64a40f4b: Received BeginTabletCopySession request for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 from peer b61b88548de84191b17de06cca3c435a ({username='slave'} at 127.14.19.132:35497)
I20250110 03:06:56.636890 17477 tablet_copy_service.cc:161] P 92bcfebb99cc47bba372938b64a40f4b: Beginning new tablet copy session on tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 from peer b61b88548de84191b17de06cca3c435a at {username='slave'} at 127.14.19.132:35497: session id = b61b88548de84191b17de06cca3c435a-83d2ceb09bc5442f8c329ca6b3ad6f35
I20250110 03:06:56.637499 17477 tablet_copy_source_session.cc:215] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.638126 19471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.639395 19471 tablet_copy_client.cc:806] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.639503 19471 tablet_copy_client.cc:670] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.640887 19471 tablet_copy_client.cc:538] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.641562 19471 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: Bootstrap starting.
I20250110 03:06:56.643307 19505 ts_tablet_manager.cc:927] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:56.643503 19505 tablet_copy_client.cc:323] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:06:56.643851 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet c8b6b5410b4e4405b419e22daf379c6a from peer d1e6d3913b04475f8962dd6496fbea0a ({username='slave'} at 127.14.19.133:40683)
I20250110 03:06:56.643926 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet c8b6b5410b4e4405b419e22daf379c6a from peer d1e6d3913b04475f8962dd6496fbea0a at {username='slave'} at 127.14.19.133:40683: session id = d1e6d3913b04475f8962dd6496fbea0a-c8b6b5410b4e4405b419e22daf379c6a
I20250110 03:06:56.644450 17608 tablet_copy_source_session.cc:215] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.645357 19505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b6b5410b4e4405b419e22daf379c6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.648008 19505 tablet_copy_client.cc:806] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.648155 19505 tablet_copy_client.cc:670] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.649015 19505 tablet_copy_client.cc:538] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.649621 19505 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap starting.
I20250110 03:06:56.664007 19457 ts_tablet_manager.cc:927] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:56.664162 19457 tablet_copy_client.cc:323] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:06:56.664857 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet fa58142d4f86446b99561d0a0fb9fc96 from peer 4ce3080cc7de4c729423b95fb7158a07 ({username='slave'} at 127.14.19.134:39939)
I20250110 03:06:56.664916 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet fa58142d4f86446b99561d0a0fb9fc96 from peer 4ce3080cc7de4c729423b95fb7158a07 at {username='slave'} at 127.14.19.134:39939: session id = 4ce3080cc7de4c729423b95fb7158a07-fa58142d4f86446b99561d0a0fb9fc96
I20250110 03:06:56.665452 17608 tablet_copy_source_session.cc:215] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.666015 19457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa58142d4f86446b99561d0a0fb9fc96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.667460 19457 tablet_copy_client.cc:806] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.667575 19457 tablet_copy_client.cc:670] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.668179 19471 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: Bootstrap replayed 1/1 log segments. Stats: ops{read=98 overwritten=0 applied=97 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:56.669981 19471 tablet_bootstrap.cc:492] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: Bootstrap complete.
I20250110 03:06:56.670243 19471 ts_tablet_manager.cc:1397] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.007s
I20250110 03:06:56.670493 19471 raft_consensus.cc:357] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.671142 19505 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=139 overwritten=0 applied=139 ignored=0} inserts{seen=811 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.671386 19505 tablet_bootstrap.cc:492] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: Bootstrap complete.
I20250110 03:06:56.671502 19471 raft_consensus.cc:738] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b61b88548de84191b17de06cca3c435a, State: Initialized, Role: LEARNER
I20250110 03:06:56.671614 19471 consensus_queue.cc:260] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.98, Last appended by leader: 98, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.672049 19471 ts_tablet_manager.cc:1428] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:06:56.672536 17477 tablet_copy_service.cc:342] P 92bcfebb99cc47bba372938b64a40f4b: Request end of tablet copy session b61b88548de84191b17de06cca3c435a-83d2ceb09bc5442f8c329ca6b3ad6f35 received from {username='slave'} at 127.14.19.132:35497
I20250110 03:06:56.672596 17477 tablet_copy_service.cc:434] P 92bcfebb99cc47bba372938b64a40f4b: ending tablet copy session b61b88548de84191b17de06cca3c435a-83d2ceb09bc5442f8c329ca6b3ad6f35 on tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 with peer b61b88548de84191b17de06cca3c435a
I20250110 03:06:56.673132 19505 ts_tablet_manager.cc:1397] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.003s
I20250110 03:06:56.673425 19505 raft_consensus.cc:357] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.673683 19505 raft_consensus.cc:738] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d1e6d3913b04475f8962dd6496fbea0a, State: Initialized, Role: LEARNER
I20250110 03:06:56.673728 19505 consensus_queue.cc:260] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 139, Last appended: 2.139, Last appended by leader: 139, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }
I20250110 03:06:56.674012 19505 ts_tablet_manager.cc:1428] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:06:56.674985 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session d1e6d3913b04475f8962dd6496fbea0a-c8b6b5410b4e4405b419e22daf379c6a received from {username='slave'} at 127.14.19.133:40683
I20250110 03:06:56.675037 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session d1e6d3913b04475f8962dd6496fbea0a-c8b6b5410b4e4405b419e22daf379c6a on tablet c8b6b5410b4e4405b419e22daf379c6a with peer d1e6d3913b04475f8962dd6496fbea0a
I20250110 03:06:56.675679 19457 tablet_copy_client.cc:538] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.676378 19457 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap starting.
I20250110 03:06:56.697568 19457 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=114 overwritten=0 applied=114 ignored=0} inserts{seen=786 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:56.697844 19457 tablet_bootstrap.cc:492] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap complete.
I20250110 03:06:56.697973 19457 ts_tablet_manager.cc:1397] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.004s
I20250110 03:06:56.698114 19457 raft_consensus.cc:357] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:56.698196 19457 raft_consensus.cc:738] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Initialized, Role: LEARNER
I20250110 03:06:56.698233 19457 consensus_queue.cc:260] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 114, Last appended: 1.114, Last appended by leader: 114, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:56.699033 19457 ts_tablet_manager.cc:1428] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:06:56.699821 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session 4ce3080cc7de4c729423b95fb7158a07-fa58142d4f86446b99561d0a0fb9fc96 received from {username='slave'} at 127.14.19.134:39939
I20250110 03:06:56.699891 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session 4ce3080cc7de4c729423b95fb7158a07-fa58142d4f86446b99561d0a0fb9fc96 on tablet fa58142d4f86446b99561d0a0fb9fc96 with peer 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:56.726353 19451 ts_tablet_manager.cc:927] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: Initiating tablet copy from peer 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:56.726526 19451 tablet_copy_client.cc:323] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42521
I20250110 03:06:56.730250 17477 tablet_copy_service.cc:140] P 92bcfebb99cc47bba372938b64a40f4b: Received BeginTabletCopySession request for tablet e365416b475a4288b17d235deddf139d from peer d28dc7130e374d8486cc68018c03fd34 ({username='slave'} at 127.14.19.135:49291)
I20250110 03:06:56.730322 17477 tablet_copy_service.cc:161] P 92bcfebb99cc47bba372938b64a40f4b: Beginning new tablet copy session on tablet e365416b475a4288b17d235deddf139d from peer d28dc7130e374d8486cc68018c03fd34 at {username='slave'} at 127.14.19.135:49291: session id = d28dc7130e374d8486cc68018c03fd34-e365416b475a4288b17d235deddf139d
I20250110 03:06:56.730897 17477 tablet_copy_source_session.cc:215] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.731689 19451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e365416b475a4288b17d235deddf139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.733139 19451 tablet_copy_client.cc:806] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.733253 19451 tablet_copy_client.cc:670] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.735481 19451 tablet_copy_client.cc:538] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.736176 19451 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: Bootstrap starting.
I20250110 03:06:56.747145 19471 ts_tablet_manager.cc:927] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: Initiating tablet copy from peer 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:06:56.747318 19471 tablet_copy_client.cc:323] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42521
I20250110 03:06:56.747521 17477 tablet_copy_service.cc:140] P 92bcfebb99cc47bba372938b64a40f4b: Received BeginTabletCopySession request for tablet f3ec6ddcb58d48d89f4cea0b3db80035 from peer b61b88548de84191b17de06cca3c435a ({username='slave'} at 127.14.19.132:35497)
I20250110 03:06:56.747586 17477 tablet_copy_service.cc:161] P 92bcfebb99cc47bba372938b64a40f4b: Beginning new tablet copy session on tablet f3ec6ddcb58d48d89f4cea0b3db80035 from peer b61b88548de84191b17de06cca3c435a at {username='slave'} at 127.14.19.132:35497: session id = b61b88548de84191b17de06cca3c435a-f3ec6ddcb58d48d89f4cea0b3db80035
I20250110 03:06:56.748114 17477 tablet_copy_source_session.cc:215] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:56.748456 19471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ec6ddcb58d48d89f4cea0b3db80035. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:56.749775 19471 tablet_copy_client.cc:806] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:56.749881 19471 tablet_copy_client.cc:670] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:56.761781 19471 tablet_copy_client.cc:538] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:56.762506 19471 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: Bootstrap starting.
I20250110 03:06:56.773699 19451 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=168 overwritten=0 applied=167 ignored=0} inserts{seen=852 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:56.773957 19451 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: Bootstrap complete.
I20250110 03:06:56.774082 19451 ts_tablet_manager.cc:1397] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.007s
I20250110 03:06:56.774214 19451 raft_consensus.cc:357] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.774787 19451 raft_consensus.cc:738] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d28dc7130e374d8486cc68018c03fd34, State: Initialized, Role: LEARNER
I20250110 03:06:56.774854 19451 consensus_queue.cc:260] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 167, Last appended: 1.168, Last appended by leader: 168, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:56.774966 19451 ts_tablet_manager.cc:1428] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 03:06:56.775195 17477 tablet_copy_service.cc:342] P 92bcfebb99cc47bba372938b64a40f4b: Request end of tablet copy session d28dc7130e374d8486cc68018c03fd34-e365416b475a4288b17d235deddf139d received from {username='slave'} at 127.14.19.135:49291
I20250110 03:06:56.775254 17477 tablet_copy_service.cc:434] P 92bcfebb99cc47bba372938b64a40f4b: ending tablet copy session d28dc7130e374d8486cc68018c03fd34-e365416b475a4288b17d235deddf139d on tablet e365416b475a4288b17d235deddf139d with peer d28dc7130e374d8486cc68018c03fd34
I20250110 03:06:56.775848 19470 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:06:56.775964 19470 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:56.776235 19470 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:56.780099 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888f2d12f3254197a6a490e23016bd46" candidate_uuid: "4ce3080cc7de4c729423b95fb7158a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59" is_pre_election: true
I20250110 03:06:56.780182 17719 raft_consensus.cc:2463] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ce3080cc7de4c729423b95fb7158a07 in term 1.
I20250110 03:06:56.780422 18902 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 4ce3080cc7de4c729423b95fb7158a07; no voters:
I20250110 03:06:56.780534 19470 raft_consensus.cc:2798] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 03:06:56.780597 19470 raft_consensus.cc:491] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:06:56.780624 19470 raft_consensus.cc:3054] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:56.781456 19470 raft_consensus.cc:513] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:56.781580 19470 leader_election.cc:290] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 2 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:56.783087 17719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888f2d12f3254197a6a490e23016bd46" candidate_uuid: "4ce3080cc7de4c729423b95fb7158a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:56.783161 17719 raft_consensus.cc:3054] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:56.783699 17719 raft_consensus.cc:2463] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce3080cc7de4c729423b95fb7158a07 in term 2.
I20250110 03:06:56.784380 18837 raft_consensus.cc:1212] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.70->[1.71-1.205] Dedup: 1.145->[1.146-1.205]
I20250110 03:06:56.786011 18902 leader_election.cc:304] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 4ce3080cc7de4c729423b95fb7158a07; no voters:
I20250110 03:06:56.786103 19470 raft_consensus.cc:2798] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:56.786190 19470 raft_consensus.cc:695] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 LEADER]: Becoming Leader. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Running, Role: LEADER
I20250110 03:06:56.786269 19470 consensus_queue.cc:237] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } }
I20250110 03:06:56.786952 17309 catalog_manager.cc:5526] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "4ce3080cc7de4c729423b95fb7158a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.788534 19471 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: Bootstrap replayed 1/1 log segments. Stats: ops{read=190 overwritten=0 applied=189 ignored=0} inserts{seen=861 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:56.788765 19471 tablet_bootstrap.cc:492] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: Bootstrap complete.
I20250110 03:06:56.788889 19471 ts_tablet_manager.cc:1397] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.011s
I20250110 03:06:56.789026 19471 raft_consensus.cc:357] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.789253 19471 raft_consensus.cc:738] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b61b88548de84191b17de06cca3c435a, State: Initialized, Role: LEARNER
I20250110 03:06:56.789301 19471 consensus_queue.cc:260] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 189, Last appended: 1.190, Last appended by leader: 190, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: true } }
I20250110 03:06:56.789407 19471 ts_tablet_manager.cc:1428] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:56.789642 17477 tablet_copy_service.cc:342] P 92bcfebb99cc47bba372938b64a40f4b: Request end of tablet copy session b61b88548de84191b17de06cca3c435a-f3ec6ddcb58d48d89f4cea0b3db80035 received from {username='slave'} at 127.14.19.132:35497
I20250110 03:06:56.789695 17477 tablet_copy_service.cc:434] P 92bcfebb99cc47bba372938b64a40f4b: ending tablet copy session b61b88548de84191b17de06cca3c435a-f3ec6ddcb58d48d89f4cea0b3db80035 on tablet f3ec6ddcb58d48d89f4cea0b3db80035 with peer b61b88548de84191b17de06cca3c435a
I20250110 03:06:56.790306 18706 raft_consensus.cc:1212] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.60->[1.61-1.163] Dedup: 1.123->[1.124-1.163]
I20250110 03:06:56.794283 18969 raft_consensus.cc:1212] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.68->[1.69-1.176] Dedup: 1.122->[1.123-1.176]
I20250110 03:06:56.800493 19319 raft_consensus.cc:1059] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59: attempting to promote NON_VOTER d1e6d3913b04475f8962dd6496fbea0a to VOTER
I20250110 03:06:56.800661 19319 consensus_queue.cc:237] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:56.801168 18837 raft_consensus.cc:1270] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20250110 03:06:56.801362 19215 consensus_queue.cc:1035] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 205, Time since last communication: 0.000s
I20250110 03:06:56.801651 19219 raft_consensus.cc:1059] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 to VOTER
I20250110 03:06:56.801831 19219 consensus_queue.cc:237] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 176, Committed index: 176, Last appended: 1.176, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:56.802382 17585 raft_consensus.cc:1270] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20250110 03:06:56.802435 19215 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 71 to 207, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.802585 17717 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20250110 03:06:56.802654 19223 consensus_queue.cc:1035] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 205, Time since last communication: 0.000s
I20250110 03:06:56.803040 18523 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20250110 03:06:56.803076 17585 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 71 to 207, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.803370 18969 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 177. (index mismatch)
I20250110 03:06:56.803910 19222 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 177, Last known committed idx: 176, Time since last communication: 0.000s
I20250110 03:06:56.804087 18837 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 71 to 207, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.804342 17309 catalog_manager.cc:5526] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 71 to 207, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.804855 19222 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.177: config changed from index 69 to 177, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:56.804976 18969 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 69 to 177, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:56.805925 17313 catalog_manager.cc:5526] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 reported cstate change: config changed from index 69 to 177, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:56.806587 17719 consensus_queue.cc:237] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:56.807257 17719 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.177: config changed from index 69 to 177, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:56.808126 18837 raft_consensus.cc:1270] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch)
I20250110 03:06:56.808319 18569 consensus_queue.cc:1035] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20250110 03:06:56.808948 19182 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index 207 to 208, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.809883 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8dec030e45a84eb89321f275d0013826 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:56.810094 17313 catalog_manager.cc:5526] T 8dec030e45a84eb89321f275d0013826 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 207 to 208, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.810107 18837 raft_consensus.cc:2949] T 8dec030e45a84eb89321f275d0013826 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index 207 to 208, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.811370 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dec030e45a84eb89321f275d0013826 with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:57906
I20250110 03:06:56.812455 19575 tablet_replica.cc:331] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:06:56.813397 19575 raft_consensus.cc:2238] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:56.813510 19348 raft_consensus.cc:1059] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b: attempting to promote NON_VOTER b61b88548de84191b17de06cca3c435a to VOTER
I20250110 03:06:56.813673 19575 raft_consensus.cc:2267] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:56.813664 19348 consensus_queue.cc:237] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 1.164, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:56.814474 18706 raft_consensus.cc:1270] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20250110 03:06:56.814846 18437 consensus_queue.cc:1035] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 163, Time since last communication: 0.000s
I20250110 03:06:56.815090 19575 ts_tablet_manager.cc:1905] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:56.815279 17719 raft_consensus.cc:1270] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 164. Preceding OpId from leader: term: 1 index: 165. (index mismatch)
I20250110 03:06:56.815421 18437 consensus_queue.cc:1035] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 163, Time since last communication: 0.000s
I20250110 03:06:56.815675 18437 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.165: config changed from index 61 to 165, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:56.815806 18706 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 61 to 165, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:56.817145 17307 catalog_manager.cc:5526] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 61 to 165, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:56.817672 19575 ts_tablet_manager.cc:1918] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.207
I20250110 03:06:56.817740 19575 log.cc:1198] T 8dec030e45a84eb89321f275d0013826 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/8dec030e45a84eb89321f275d0013826
I20250110 03:06:56.818266 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet 8dec030e45a84eb89321f275d0013826 (table rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]) successfully deleted
I20250110 03:06:56.818421 17455 consensus_queue.cc:237] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 1.166, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:56.818787 18706 raft_consensus.cc:1270] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch)
I20250110 03:06:56.818763 17719 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.165: config changed from index 61 to 165, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:56.818953 18437 consensus_queue.cc:1035] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s
I20250110 03:06:56.819473 19348 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.167: config changed from index 165 to 167, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:56.820546 17299 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f8b5177de1fa4e1c9668453452f2064f with cas_config_opid_index 165: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:56.820709 17313 catalog_manager.cc:5526] T f8b5177de1fa4e1c9668453452f2064f P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index 165 to 167, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.821020 18706 raft_consensus.cc:2949] T f8b5177de1fa4e1c9668453452f2064f P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 165 to 167, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:56.821291 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet f8b5177de1fa4e1c9668453452f2064f with delete_type TABLET_DATA_TOMBSTONED (TS 0b345888171f4abcb611a51e27cbcd59 not found in new config with opid_index 167) from {username='slave'} at 127.0.0.1:40906
I20250110 03:06:56.821607 19576 tablet_replica.cc:331] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: stopping tablet replica
I20250110 03:06:56.821753 19576 raft_consensus.cc:2238] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:56.821866 19576 raft_consensus.cc:2267] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:56.822314 19576 ts_tablet_manager.cc:1905] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:56.827911 19576 ts_tablet_manager.cc:1918] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.165
I20250110 03:06:56.828051 19576 log.cc:1198] T f8b5177de1fa4e1c9668453452f2064f P 0b345888171f4abcb611a51e27cbcd59: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8b5177de1fa4e1c9668453452f2064f
I20250110 03:06:56.828615 17300 catalog_manager.cc:4872] TS 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605): tablet f8b5177de1fa4e1c9668453452f2064f (table rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]) successfully deleted
I20250110 03:06:56.874801 19104 raft_consensus.cc:1212] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.189] Dedup: 1.123->[1.124-1.189]
I20250110 03:06:56.887356 19319 raft_consensus.cc:1059] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59: attempting to promote NON_VOTER d28dc7130e374d8486cc68018c03fd34 to VOTER
I20250110 03:06:56.887516 19319 consensus_queue.cc:237] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 190, Committed index: 190, Last appended: 1.190, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:56.887881 17455 raft_consensus.cc:1270] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 190. Preceding OpId from leader: term: 1 index: 191. (index mismatch)
I20250110 03:06:56.888175 18569 consensus_queue.cc:1035] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 191, Last known committed idx: 190, Time since last communication: 0.000s
I20250110 03:06:56.889209 19104 raft_consensus.cc:1270] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 190. Preceding OpId from leader: term: 1 index: 191. (index mismatch)
I20250110 03:06:56.889346 18569 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.191: config changed from index 66 to 191, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.889569 19215 consensus_queue.cc:1035] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 191, Last known committed idx: 190, Time since last communication: 0.000s
I20250110 03:06:56.890875 19104 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Committing config change with OpId 1.191: config changed from index 66 to 191, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.891165 17313 catalog_manager.cc:5526] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 66 to 191, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:56.891499 17455 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.191: config changed from index 66 to 191, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 191 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.893169 19103 raft_consensus.cc:1212] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.220] Dedup: 1.162->[1.163-1.220]
I20250110 03:06:56.896660 17716 consensus_queue.cc:237] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 192, Committed index: 192, Last appended: 1.193, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:56.897257 19103 raft_consensus.cc:1270] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 193. Preceding OpId from leader: term: 1 index: 194. (index mismatch)
I20250110 03:06:56.897477 19182 consensus_queue.cc:1035] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 194, Last known committed idx: 192, Time since last communication: 0.000s
I20250110 03:06:56.900269 19215 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.194: config changed from index 191 to 194, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.900378 19104 raft_consensus.cc:2949] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Committing config change with OpId 1.194: config changed from index 191 to 194, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.901298 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8b2221475806475c83aa9867a548f51f with cas_config_opid_index 191: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:56.901355 17307 catalog_manager.cc:5526] T 8b2221475806475c83aa9867a548f51f P d28dc7130e374d8486cc68018c03fd34 reported cstate change: config changed from index 191 to 194, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 194 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.902288 18569 raft_consensus.cc:1059] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: attempting to promote NON_VOTER d28dc7130e374d8486cc68018c03fd34 to VOTER
I20250110 03:06:56.902407 18569 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:56.903025 19103 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20250110 03:06:56.903115 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b2221475806475c83aa9867a548f51f with delete_type TABLET_DATA_TOMBSTONED (TS 92bcfebb99cc47bba372938b64a40f4b not found in new config with opid_index 194) from {username='slave'} at 127.0.0.1:60766
I20250110 03:06:56.903194 19223 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20250110 03:06:56.903381 19619 tablet_replica.cc:331] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: stopping tablet replica
I20250110 03:06:56.903831 19223 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 79 to 222, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.904418 17587 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20250110 03:06:56.905073 19103 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 79 to 222, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.905637 18569 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20250110 03:06:56.905645 17313 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 79 to 222, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.906244 17587 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 79 to 222, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:56.903740 19619 raft_consensus.cc:2238] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:56.906738 19619 raft_consensus.cc:2267] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:56.907135 19619 ts_tablet_manager.cc:1905] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:56.909956 19619 ts_tablet_manager.cc:1918] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.192
I20250110 03:06:56.910030 19619 log.cc:1198] T 8b2221475806475c83aa9867a548f51f P 92bcfebb99cc47bba372938b64a40f4b: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b2221475806475c83aa9867a548f51f
I20250110 03:06:56.910506 17299 catalog_manager.cc:4872] TS 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521): tablet 8b2221475806475c83aa9867a548f51f (table rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]) successfully deleted
I20250110 03:06:56.922446 17587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "567885520d33460fae980ae73fabcd6a"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58202
I20250110 03:06:56.922536 17587 raft_consensus.cc:604] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:56.922618 19222 raft_consensus.cc:988] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: : Instructing follower 4ce3080cc7de4c729423b95fb7158a07 to start an election
I20250110 03:06:56.922665 19222 raft_consensus.cc:1076] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Signalling peer 4ce3080cc7de4c729423b95fb7158a07 to start an election
I20250110 03:06:56.922950 18969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "567885520d33460fae980ae73fabcd6a"
dest_uuid: "4ce3080cc7de4c729423b95fb7158a07"
from {username='slave'} at 127.14.19.130:36461
I20250110 03:06:56.923045 18969 raft_consensus.cc:491] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:06:56.923077 18969 raft_consensus.cc:3054] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:56.923826 18969 raft_consensus.cc:513] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:56.923990 18969 leader_election.cc:290] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 2 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605), 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:56.924157 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567885520d33460fae980ae73fabcd6a" candidate_uuid: "4ce3080cc7de4c729423b95fb7158a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 197 } } ignore_live_leader: true dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:56.924228 17718 raft_consensus.cc:3054] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:56.925020 17718 raft_consensus.cc:2463] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce3080cc7de4c729423b95fb7158a07 in term 2.
I20250110 03:06:56.925276 18902 leader_election.cc:304] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 4ce3080cc7de4c729423b95fb7158a07; no voters:
I20250110 03:06:56.925382 19560 raft_consensus.cc:2798] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:56.925432 19560 raft_consensus.cc:695] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 2 LEADER]: Becoming Leader. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Running, Role: LEADER
I20250110 03:06:56.925474 19560 consensus_queue.cc:237] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 1.197, Last appended by leader: 197, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
W20250110 03:06:56.925817 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:56.925897 17307 catalog_manager.cc:5526] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 reported cstate change: term changed from 1 to 2, leader changed from 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) to 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134). New cstate: current_term: 2 leader_uuid: "4ce3080cc7de4c729423b95fb7158a07" committed_config { opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.927187 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567885520d33460fae980ae73fabcd6a" candidate_uuid: "4ce3080cc7de4c729423b95fb7158a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 197 } } ignore_live_leader: true dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:56.927263 17587 raft_consensus.cc:3049] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:06:56.927261 17718 raft_consensus.cc:1235] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Rejecting Update request from peer 74fcf6c6434b49d8a777aaeda45d971e for earlier term 1. Current term is 2. Ops: [1.197-1.197]
I20250110 03:06:56.927289 17587 raft_consensus.cc:738] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:56.927332 17587 consensus_queue.cc:260] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 1.197, Last appended by leader: 197, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 177 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:56.927407 17587 raft_consensus.cc:3054] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:56.928145 17587 raft_consensus.cc:2463] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ce3080cc7de4c729423b95fb7158a07 in term 2.
W20250110 03:06:56.929409 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Illegal state: replica 74fcf6c6434b49d8a777aaeda45d971e is not leader of this config: current role FOLLOWER
I20250110 03:06:56.929453 18837 raft_consensus.cc:1212] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.211] Dedup: 1.172->[1.173-1.211]
W20250110 03:06:56.940227 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60210: Illegal state: replica 0b345888171f4abcb611a51e27cbcd59 is not leader of this config: current role FOLLOWER
I20250110 03:06:56.952693 17585 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Refusing update from remote peer 4ce3080cc7de4c729423b95fb7158a07: Log matching property violated. Preceding OpId in replica: term: 1 index: 197. Preceding OpId from leader: term: 2 index: 199. (index mismatch)
I20250110 03:06:56.952868 17717 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer 4ce3080cc7de4c729423b95fb7158a07: Log matching property violated. Preceding OpId in replica: term: 1 index: 196. Preceding OpId from leader: term: 2 index: 199. (index mismatch)
I20250110 03:06:56.953068 19470 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s
I20250110 03:06:56.953235 19470 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 196, Time since last communication: 0.000s
I20250110 03:06:56.954655 19469 mvcc.cc:204] Tried to move back new op lower bound from 7112615595836092416 to 7112615595727126528. Current Snapshot: MvccSnapshot[applied={T|T < 7112615595836092416 or (T in {7112615595836092416})}]
I20250110 03:06:56.961352 19219 raft_consensus.cc:1059] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER d1e6d3913b04475f8962dd6496fbea0a to VOTER
I20250110 03:06:56.961475 19219 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 211, Committed index: 211, Last appended: 1.212, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:56.962349 17718 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20250110 03:06:56.962343 18837 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 212. Preceding OpId from leader: term: 1 index: 213. (index mismatch)
I20250110 03:06:56.962620 18618 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20250110 03:06:56.963223 18618 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 211, Time since last communication: 0.000s
I20250110 03:06:56.963176 18523 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.213: config changed from index 96 to 213, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.963562 17716 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 96 to 213, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.963726 18969 consensus_queue.cc:237] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 200, Committed index: 200, Last appended: 2.200, Last appended by leader: 197, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:56.964284 17718 raft_consensus.cc:1270] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer 4ce3080cc7de4c729423b95fb7158a07: Log matching property violated. Preceding OpId in replica: term: 2 index: 200. Preceding OpId from leader: term: 2 index: 201. (index mismatch)
I20250110 03:06:56.964560 19560 consensus_queue.cc:1035] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 201, Last known committed idx: 200, Time since last communication: 0.000s
I20250110 03:06:56.964534 17307 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 96 to 213, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.965879 18837 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.213: config changed from index 96 to 213, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:56.966346 19470 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 [term 2 LEADER]: Committing config change with OpId 2.201: config changed from index 177 to 201, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:56.967677 17299 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 567885520d33460fae980ae73fabcd6a with cas_config_opid_index 177: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:56.968178 17716 raft_consensus.cc:2949] T 567885520d33460fae980ae73fabcd6a P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Committing config change with OpId 2.201: config changed from index 177 to 201, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:56.968181 17313 catalog_manager.cc:5526] T 567885520d33460fae980ae73fabcd6a P 4ce3080cc7de4c729423b95fb7158a07 reported cstate change: config changed from index 177 to 201, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "4ce3080cc7de4c729423b95fb7158a07" committed_config { opid_index: 201 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:56.968920 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 567885520d33460fae980ae73fabcd6a with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 201) from {username='slave'} at 127.0.0.1:57906
I20250110 03:06:56.969020 19575 tablet_replica.cc:331] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:06:56.969080 19575 raft_consensus.cc:2238] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:56.969123 19575 raft_consensus.cc:2267] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:56.969548 19575 ts_tablet_manager.cc:1905] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:56.971001 19575 ts_tablet_manager.cc:1918] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.200
I20250110 03:06:56.971055 19575 log.cc:1198] T 567885520d33460fae980ae73fabcd6a P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/567885520d33460fae980ae73fabcd6a
I20250110 03:06:56.971640 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet 567885520d33460fae980ae73fabcd6a (table rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]) successfully deleted
I20250110 03:06:56.979983 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60242
I20250110 03:06:56.980067 17718 raft_consensus.cc:604] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:56.988188 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:56.992785 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:56.998538 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.006611 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.015041 17585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85fe921f114943c896a041599c39dbe5"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58230
I20250110 03:06:57.015122 17585 raft_consensus.cc:604] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:57.015430 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.017818 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.019304 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.022805 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.022975 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.027618 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.029313 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.033041 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.033857 18706 raft_consensus.cc:1212] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.81->[1.82-1.240] Dedup: 1.155->[1.156-1.240]
I20250110 03:06:57.038410 19171 raft_consensus.cc:1059] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: attempting to promote NON_VOTER b61b88548de84191b17de06cca3c435a to VOTER
I20250110 03:06:57.038585 19171 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.241, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:57.039110 17718 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20250110 03:06:57.039304 19212 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 240, Time since last communication: 0.000s
W20250110 03:06:57.039453 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.039685 18706 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch)
I20250110 03:06:57.039870 19212 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 240, Time since last communication: 0.000s
I20250110 03:06:57.040648 18437 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.242: config changed from index 82 to 242, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.041570 18705 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 82 to 242, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
W20250110 03:06:57.041774 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.041800 17718 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 82 to 242, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.042762 17313 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 82 to 242, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.043849 17455 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.243, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:57.044597 18705 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 1 index: 244. (index mismatch)
I20250110 03:06:57.044929 19318 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 242, Time since last communication: 0.000s
I20250110 03:06:57.046171 18705 raft_consensus.cc:1212] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.242->[1.243-1.244] Dedup: 1.243->[1.244-1.244]
I20250110 03:06:57.046633 19171 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.244: config changed from index 242 to 244, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.047376 18705 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.244: config changed from index 242 to 244, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
W20250110 03:06:57.047732 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.047899 17299 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 593fb2f6a0a7417ca54b800249318eae with cas_config_opid_index 242: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.048492 17314 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 242 to 244, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.049324 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet 593fb2f6a0a7417ca54b800249318eae with delete_type TABLET_DATA_TOMBSTONED (TS 0b345888171f4abcb611a51e27cbcd59 not found in new config with opid_index 244) from {username='slave'} at 127.0.0.1:40906
I20250110 03:06:57.049427 19576 tablet_replica.cc:331] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: stopping tablet replica
I20250110 03:06:57.049486 19576 raft_consensus.cc:2238] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.049520 19576 pending_rounds.cc:62] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Trying to abort 1 pending ops.
I20250110 03:06:57.049566 19576 pending_rounds.cc:69] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Aborting op as it isn't in flight: id { term: 1 index: 243 } timestamp: 7112615596204199936 op_type: WRITE_OP write_request { tablet_id: "593fb2f6a0a7417ca54b800249318eae" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000:N\020@$\013\374\215\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""2b5dc70db36ebb19" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7112615596182798336 authz_token { token_data: "\010\373\237\202\274\006\"3\n\005slave\022*\n 547950e4f5174f7c8de141cb25c6be98\020\001\030\001 \001(\001" signature: "<redacted>""c\265\013\013e\\\201r\207\361\276\247\025L\302\\\001S\277uz\377\242\330>G\263\257Q\'\224\036\251\243\225\014KV\261/N\2325SG\321\300R\222\331d\277\321\\F\346\t\207<\000\353\371c\303" signing_key_seq_num: 0 } } request_id { client_id: "a936c49d279547b3ac102e7a7a6a2b28" seq_no: 639 first_incomplete_seq_no: 639 attempt_no: 0 }
I20250110 03:06:57.049695 19576 raft_consensus.cc:2267] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.050899 19576 ts_tablet_manager.cc:1905] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:57.052510 19576 ts_tablet_manager.cc:1918] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.243
I20250110 03:06:57.052567 19576 log.cc:1198] T 593fb2f6a0a7417ca54b800249318eae P 0b345888171f4abcb611a51e27cbcd59: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/593fb2f6a0a7417ca54b800249318eae
I20250110 03:06:57.052928 17300 catalog_manager.cc:4872] TS 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605): tablet 593fb2f6a0a7417ca54b800249318eae (table rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]) successfully deleted
W20250110 03:06:57.052958 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.059183 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.059969 17716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60262
I20250110 03:06:57.060039 17716 raft_consensus.cc:604] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:57.062356 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.068958 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.073843 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.078648 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.080448 17585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85fe921f114943c896a041599c39dbe5"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58276
I20250110 03:06:57.080524 17585 raft_consensus.cc:604] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:57.086262 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.092094 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.100132 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.100239 18837 raft_consensus.cc:1212] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.91->[1.92-1.120] Dedup: 1.112->[1.113-1.120]
W20250110 03:06:57.105528 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.116489 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60294
I20250110 03:06:57.116583 17718 raft_consensus.cc:604] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:57.118212 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.119311 18837 raft_consensus.cc:1212] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.61->[2.62-2.213] Dedup: 2.139->[2.140-2.213]
W20250110 03:06:57.121978 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.126060 18522 raft_consensus.cc:1059] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER d1e6d3913b04475f8962dd6496fbea0a to VOTER
I20250110 03:06:57.126726 18522 consensus_queue.cc:237] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.126425 18523 raft_consensus.cc:1059] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER d1e6d3913b04475f8962dd6496fbea0a to VOTER
I20250110 03:06:57.127110 18837 raft_consensus.cc:1270] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
W20250110 03:06:57.127147 18523 raft_consensus.cc:1062] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e: Unable to promote non-voter d1e6d3913b04475f8962dd6496fbea0a: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: true } }.
Pending config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.127254 17455 raft_consensus.cc:1270] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20250110 03:06:57.127470 18523 consensus_queue.cc:1035] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20250110 03:06:57.127737 18523 consensus_queue.cc:1035] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20250110 03:06:57.128559 19219 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 62 to 215, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.129652 17307 catalog_manager.cc:5526] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 62 to 215, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.130255 18837 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 62 to 215, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.130707 17585 consensus_queue.cc:237] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 2.216, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.130976 17455 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 62 to 215, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.131361 18837 raft_consensus.cc:1270] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 215. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20250110 03:06:57.131613 18522 consensus_queue.cc:1035] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 215, Time since last communication: 0.000s
I20250110 03:06:57.132462 18706 raft_consensus.cc:1212] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.88->[1.89-1.112] Dedup: 1.98->[1.99-1.112]
I20250110 03:06:57.132566 19222 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.217: config changed from index 215 to 217, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.133297 18837 raft_consensus.cc:2949] T c8b6b5410b4e4405b419e22daf379c6a P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Committing config change with OpId 2.217: config changed from index 215 to 217, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.133574 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c8b6b5410b4e4405b419e22daf379c6a with cas_config_opid_index 215: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.134503 17314 catalog_manager.cc:5526] T c8b6b5410b4e4405b419e22daf379c6a P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 215 to 217, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.135185 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet c8b6b5410b4e4405b419e22daf379c6a with delete_type TABLET_DATA_TOMBSTONED (TS 92bcfebb99cc47bba372938b64a40f4b not found in new config with opid_index 217) from {username='slave'} at 127.0.0.1:60766
I20250110 03:06:57.135373 19619 tablet_replica.cc:331] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: stopping tablet replica
I20250110 03:06:57.135432 19619 raft_consensus.cc:2238] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.135469 19619 raft_consensus.cc:2267] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.136363 19619 ts_tablet_manager.cc:1905] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:57.137691 19619 ts_tablet_manager.cc:1918] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.215
I20250110 03:06:57.137739 19619 log.cc:1198] T c8b6b5410b4e4405b419e22daf379c6a P 92bcfebb99cc47bba372938b64a40f4b: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/c8b6b5410b4e4405b419e22daf379c6a
W20250110 03:06:57.138154 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.138372 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.138579 17299 catalog_manager.cc:4872] TS 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521): tablet c8b6b5410b4e4405b419e22daf379c6a (table rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]) successfully deleted
I20250110 03:06:57.146168 17587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85fe921f114943c896a041599c39dbe5"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58298
I20250110 03:06:57.146236 17587 raft_consensus.cc:604] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:57.154886 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.156049 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.172291 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.173420 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.182878 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60312
I20250110 03:06:57.182953 17718 raft_consensus.cc:604] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:57.190745 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.191390 18969 raft_consensus.cc:1212] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.107->[1.108-1.127] Dedup: 1.114->[1.115-1.127]
W20250110 03:06:57.194012 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.213137 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.213493 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.235703 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.236256 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.255754 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.259156 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.265925 18618 raft_consensus.cc:1059] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 to VOTER
I20250110 03:06:57.266077 18618 consensus_queue.cc:237] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.127, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:57.266455 18969 raft_consensus.cc:1270] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20250110 03:06:57.266664 18523 consensus_queue.cc:1035] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250110 03:06:57.267184 17455 raft_consensus.cc:1270] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20250110 03:06:57.267475 18522 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.128: config changed from index 108 to 128, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:57.267889 19222 consensus_queue.cc:1035] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20250110 03:06:57.268033 18969 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 108 to 128, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:57.268172 17455 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 108 to 128, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:57.268908 17314 catalog_manager.cc:5526] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 108 to 128, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.269130 19212 raft_consensus.cc:1059] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b: attempting to promote NON_VOTER b61b88548de84191b17de06cca3c435a to VOTER
I20250110 03:06:57.269253 19212 consensus_queue.cc:237] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:57.269814 17585 raft_consensus.cc:1270] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20250110 03:06:57.270074 18706 raft_consensus.cc:1270] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20250110 03:06:57.270130 17587 consensus_queue.cc:237] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:57.270303 19171 consensus_queue.cc:1035] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20250110 03:06:57.270526 18969 raft_consensus.cc:1270] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20250110 03:06:57.270726 18523 consensus_queue.cc:1035] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20250110 03:06:57.270735 19216 consensus_queue.cc:1035] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 112, Time since last communication: 0.000s
I20250110 03:06:57.271399 18437 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index 89 to 114, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.271606 18706 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 89 to 114, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.271736 17587 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 89 to 114, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.272739 17307 catalog_manager.cc:5526] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 89 to 114, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.273388 19222 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index 128 to 129, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:57.273661 18969 raft_consensus.cc:2949] T fa58142d4f86446b99561d0a0fb9fc96 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 128 to 129, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:57.274250 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fa58142d4f86446b99561d0a0fb9fc96 with cas_config_opid_index 128: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.274253 17455 consensus_queue.cc:237] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 114, Committed index: 114, Last appended: 1.114, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:57.274664 17314 catalog_manager.cc:5526] T fa58142d4f86446b99561d0a0fb9fc96 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 128 to 129, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.275003 18704 raft_consensus.cc:1270] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20250110 03:06:57.275241 19171 consensus_queue.cc:1035] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 114, Time since last communication: 0.000s
I20250110 03:06:57.275332 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet fa58142d4f86446b99561d0a0fb9fc96 with delete_type TABLET_DATA_TOMBSTONED (TS 92bcfebb99cc47bba372938b64a40f4b not found in new config with opid_index 129) from {username='slave'} at 127.0.0.1:60766
I20250110 03:06:57.275465 19619 tablet_replica.cc:331] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: stopping tablet replica
I20250110 03:06:57.275537 19619 raft_consensus.cc:2238] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.275761 19619 raft_consensus.cc:2267] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.276093 19619 ts_tablet_manager.cc:1905] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250110 03:06:57.276240 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.277338 19619 ts_tablet_manager.cc:1918] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.128
I20250110 03:06:57.277401 19619 log.cc:1198] T fa58142d4f86446b99561d0a0fb9fc96 P 92bcfebb99cc47bba372938b64a40f4b: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa58142d4f86446b99561d0a0fb9fc96
I20250110 03:06:57.277801 17299 catalog_manager.cc:4872] TS 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521): tablet fa58142d4f86446b99561d0a0fb9fc96 (table rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]) successfully deleted
I20250110 03:06:57.277915 19216 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.115: config changed from index 114 to 115, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.278129 18705 raft_consensus.cc:2949] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 114 to 115, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.278774 17299 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 with cas_config_opid_index 114: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.279086 17307 catalog_manager.cc:5526] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index 114 to 115, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.279753 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 115) from {username='slave'} at 127.0.0.1:57906
I20250110 03:06:57.279835 19575 tablet_replica.cc:331] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:06:57.279883 19575 raft_consensus.cc:2238] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.279922 19575 raft_consensus.cc:2267] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.280184 19575 ts_tablet_manager.cc:1905] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250110 03:06:57.280565 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.281510 19575 ts_tablet_manager.cc:1918] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.114
I20250110 03:06:57.281569 19575 log.cc:1198] T 83d2ceb09bc5442f8c329ca6b3ad6f35 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/83d2ceb09bc5442f8c329ca6b3ad6f35
I20250110 03:06:57.281929 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet 83d2ceb09bc5442f8c329ca6b3ad6f35 (table rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]) successfully deleted
I20250110 03:06:57.284526 18522 raft_consensus.cc:1059] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER d1e6d3913b04475f8962dd6496fbea0a to VOTER
I20250110 03:06:57.284639 18522 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.285097 18837 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20250110 03:06:57.285346 18618 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20250110 03:06:57.285495 17455 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20250110 03:06:57.285643 18618 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20250110 03:06:57.285986 18618 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 92 to 122, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.286258 18837 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 92 to 122, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.286455 17455 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 92 to 122, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.288074 19103 raft_consensus.cc:1212] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.79->[1.80-1.250] Dedup: 1.168->[1.169-1.250]
I20250110 03:06:57.288650 17307 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a reported cstate change: config changed from index 92 to 122, d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.292104 17585 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.292604 18837 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20250110 03:06:57.292858 18523 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20250110 03:06:57.293548 18523 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 122 to 123, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.294385 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7906c51301244bd4a592d8e01e6ad602 with cas_config_opid_index 122: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.294592 19212 raft_consensus.cc:1059] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: attempting to promote NON_VOTER d28dc7130e374d8486cc68018c03fd34 to VOTER
I20250110 03:06:57.294709 19212 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.294904 18837 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 122 to 123, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.295116 19104 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20250110 03:06:57.295163 17314 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 122 to 123, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.295275 19216 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 250, Time since last communication: 0.000s
I20250110 03:06:57.295750 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet 7906c51301244bd4a592d8e01e6ad602 with delete_type TABLET_DATA_TOMBSTONED (TS 92bcfebb99cc47bba372938b64a40f4b not found in new config with opid_index 123) from {username='slave'} at 127.0.0.1:60766
I20250110 03:06:57.295832 19619 tablet_replica.cc:331] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: stopping tablet replica
I20250110 03:06:57.295879 19619 raft_consensus.cc:2238] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.295921 19619 raft_consensus.cc:2267] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.296216 19619 ts_tablet_manager.cc:1905] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:57.297497 19619 ts_tablet_manager.cc:1918] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.122
I20250110 03:06:57.297540 19619 log.cc:1198] T 7906c51301244bd4a592d8e01e6ad602 P 92bcfebb99cc47bba372938b64a40f4b: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/7906c51301244bd4a592d8e01e6ad602
I20250110 03:06:57.297837 17299 catalog_manager.cc:4872] TS 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521): tablet 7906c51301244bd4a592d8e01e6ad602 (table rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]) successfully deleted
I20250110 03:06:57.297941 17718 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch)
I20250110 03:06:57.298137 18437 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 250, Time since last communication: 0.000s
W20250110 03:06:57.298645 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
I20250110 03:06:57.299199 17718 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 80 to 252, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.299139 19216 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 80 to 252, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.301452 17307 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 80 to 252, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.302451 19104 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 80 to 252, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
W20250110 03:06:57.305987 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.324128 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.330441 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.347105 18705 raft_consensus.cc:1212] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.90->[1.91-1.214] Dedup: 1.190->[1.191-1.214]
W20250110 03:06:57.349895 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.357857 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.378562 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.383260 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.403172 17455 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32914
I20250110 03:06:57.403273 17455 raft_consensus.cc:604] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:57.404059 19212 raft_consensus.cc:988] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: : Instructing follower 0b345888171f4abcb611a51e27cbcd59 to start an election
I20250110 03:06:57.404130 19212 raft_consensus.cc:1076] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Signalling peer 0b345888171f4abcb611a51e27cbcd59 to start an election
I20250110 03:06:57.404575 17716 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
from {username='slave'} at 127.14.19.129:41183
I20250110 03:06:57.404661 17716 raft_consensus.cc:491] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:06:57.404695 17716 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:57.405551 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.405779 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
I20250110 03:06:57.407768 17718 raft_consensus.cc:1270] T 888f2d12f3254197a6a490e23016bd46 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer 4ce3080cc7de4c729423b95fb7158a07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250110 03:06:57.407969 19560 consensus_queue.cc:1035] T 888f2d12f3254197a6a490e23016bd46 P 4ce3080cc7de4c729423b95fb7158a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 03:06:57.411346 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
W20250110 03:06:57.411659 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.413834 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
I20250110 03:06:57.414683 17585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "85fe921f114943c896a041599c39dbe5"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58352
I20250110 03:06:57.414764 17585 raft_consensus.cc:604] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:57.418336 18523 raft_consensus.cc:988] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: : Instructing follower d1e6d3913b04475f8962dd6496fbea0a to start an election
I20250110 03:06:57.418407 18523 raft_consensus.cc:1076] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Signalling peer d1e6d3913b04475f8962dd6496fbea0a to start an election
I20250110 03:06:57.418678 18837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "85fe921f114943c896a041599c39dbe5"
dest_uuid: "d1e6d3913b04475f8962dd6496fbea0a"
from {username='slave'} at 127.14.19.130:38643
I20250110 03:06:57.418783 18837 raft_consensus.cc:491] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:06:57.418860 18837 raft_consensus.cc:3054] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:57.420233 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
W20250110 03:06:57.424593 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
W20250110 03:06:57.431908 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
W20250110 03:06:57.431939 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Service unavailable: leader transfer in progress
W20250110 03:06:57.439180 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
W20250110 03:06:57.441329 17415 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
I20250110 03:06:57.446625 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60328
I20250110 03:06:57.446707 17718 raft_consensus.cc:604] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:57.451694 17716 raft_consensus.cc:513] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.451853 17716 leader_election.cc:290] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521), d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779)
I20250110 03:06:57.452108 19104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365416b475a4288b17d235deddf139d" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 2 candidate_status { last_received { term: 1 index: 265 } } ignore_live_leader: true dest_uuid: "d28dc7130e374d8486cc68018c03fd34"
I20250110 03:06:57.452205 19104 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:57.452253 18837 raft_consensus.cc:513] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.452533 18837 leader_election.cc:290] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [CANDIDATE]: Term 2 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185), 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:57.453109 19104 raft_consensus.cc:2463] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 2.
W20250110 03:06:57.453313 17417 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Service unavailable: leader transfer in progress
I20250110 03:06:57.453311 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fe921f114943c896a041599c39dbe5" candidate_uuid: "d1e6d3913b04475f8962dd6496fbea0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: true dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:57.453377 17716 raft_consensus.cc:3054] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:57.453431 17655 leader_election.cc:304] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, d28dc7130e374d8486cc68018c03fd34; no voters:
I20250110 03:06:57.453503 19319 raft_consensus.cc:2798] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:57.453538 19319 raft_consensus.cc:695] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Becoming Leader. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:57.453576 19319 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.265, Last appended by leader: 265, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.454154 17716 raft_consensus.cc:2463] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e6d3913b04475f8962dd6496fbea0a in term 2.
I20250110 03:06:57.454468 18771 leader_election.cc:304] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, d1e6d3913b04475f8962dd6496fbea0a; no voters:
I20250110 03:06:57.454609 19449 raft_consensus.cc:2798] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:57.454725 19449 raft_consensus.cc:695] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEADER]: Becoming Leader. State: Replica: d1e6d3913b04475f8962dd6496fbea0a, State: Running, Role: LEADER
I20250110 03:06:57.454792 19449 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.457309 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365416b475a4288b17d235deddf139d" candidate_uuid: "0b345888171f4abcb611a51e27cbcd59" candidate_term: 2 candidate_status { last_received { term: 1 index: 265 } } ignore_live_leader: true dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:06:57.457389 17455 raft_consensus.cc:3049] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:06:57.457417 17455 raft_consensus.cc:738] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:06:57.457469 17455 consensus_queue.cc:260] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 265, Committed index: 265, Last appended: 1.265, Last appended by leader: 265, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.457556 17455 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:57.458536 17314 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: term changed from 1 to 2, leader changed from 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) to 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131). New cstate: current_term: 2 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:57.459096 17456 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32940
I20250110 03:06:57.459307 17307 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a reported cstate change: term changed from 1 to 2, leader changed from 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) to d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133). New cstate: current_term: 2 leader_uuid: "d1e6d3913b04475f8962dd6496fbea0a" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.459345 17585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fe921f114943c896a041599c39dbe5" candidate_uuid: "d1e6d3913b04475f8962dd6496fbea0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 224 } } ignore_live_leader: true dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:57.459414 17585 raft_consensus.cc:3049] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:06:57.459435 17585 raft_consensus.cc:738] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:57.459470 17585 consensus_queue.cc:260] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 224, Committed index: 224, Last appended: 1.224, Last appended by leader: 224, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.459537 17585 raft_consensus.cc:3054] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:57.461457 17547 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57980: Illegal state: replica 74fcf6c6434b49d8a777aaeda45d971e is not leader of this config: current role FOLLOWER
W20250110 03:06:57.464673 17417 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60840: Illegal state: replica 92bcfebb99cc47bba372938b64a40f4b is not leader of this config: current role FOLLOWER
W20250110 03:06:57.471707 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.474519 19104 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20250110 03:06:57.474723 19319 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 266, Last known committed idx: 265, Time since last communication: 0.000s
I20250110 03:06:57.475999 19644 mvcc.cc:204] Tried to move back new op lower bound from 7112615597974405120 to 7112615597890314240. Current Snapshot: MvccSnapshot[applied={T|T < 7112615597974405120}]
I20250110 03:06:57.476050 17716 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 2.267, Last appended by leader: 265, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.476485 19104 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 2 index: 267. Preceding OpId from leader: term: 2 index: 268. (index mismatch)
I20250110 03:06:57.476680 19215 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20250110 03:06:57.477349 18569 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Committing config change with OpId 2.268: config changed from index 252 to 268, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.477473 19104 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.268: config changed from index 252 to 268, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.485419 19319 raft_consensus.cc:988] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: : Instructing follower 74fcf6c6434b49d8a777aaeda45d971e to start an election
I20250110 03:06:57.485483 19319 raft_consensus.cc:1076] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Signalling peer 74fcf6c6434b49d8a777aaeda45d971e to start an election
I20250110 03:06:57.485692 17587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
from {username='slave'} at 127.14.19.131:50761
I20250110 03:06:57.485766 17587 raft_consensus.cc:491] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:06:57.485801 17587 raft_consensus.cc:3054] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:57.490926 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60210: Illegal state: replica 0b345888171f4abcb611a51e27cbcd59 is not leader of this config: current role FOLLOWER
W20250110 03:06:57.505247 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.525563 17716 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20250110 03:06:57.525779 19449 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 225, Last known committed idx: 224, Time since last communication: 0.000s
I20250110 03:06:57.526787 19448 mvcc.cc:204] Tried to move back new op lower bound from 7112615598183186432 to 7112615597895315456. Current Snapshot: MvccSnapshot[applied={T|T < 7112615598183186432}]
I20250110 03:06:57.531315 18837 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 2.227, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } }
I20250110 03:06:57.531633 17716 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20250110 03:06:57.531788 19901 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20250110 03:06:57.532312 19449 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEADER]: Committing config change with OpId 2.228: config changed from index 213 to 228, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.532457 17716 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Committing config change with OpId 2.228: config changed from index 213 to 228, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
W20250110 03:06:57.539819 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Service unavailable: leader transfer in progress
I20250110 03:06:57.552995 17585 raft_consensus.cc:2463] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1e6d3913b04475f8962dd6496fbea0a in term 2.
I20250110 03:06:57.552991 17455 raft_consensus.cc:2463] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b345888171f4abcb611a51e27cbcd59 in term 2.
I20250110 03:06:57.553118 17457 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 265. Preceding OpId from leader: term: 2 index: 267. (index mismatch)
I20250110 03:06:57.553416 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e365416b475a4288b17d235deddf139d with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.553723 17587 raft_consensus.cc:513] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.553838 17456 raft_consensus.cc:604] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Received request to transfer leadership
I20250110 03:06:57.553882 17456 raft_consensus.cc:612] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250110 03:06:57.553879 17587 leader_election.cc:290] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605), d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779)
I20250110 03:06:57.553947 17307 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 252 to 268, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:57.554654 17588 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 224. Preceding OpId from leader: term: 2 index: 226. (index mismatch)
I20250110 03:06:57.555044 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 2 candidate_status { last_received { term: 1 index: 231 } } ignore_live_leader: true dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:57.555112 17718 raft_consensus.cc:3049] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:06:57.555159 17718 raft_consensus.cc:738] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:57.555220 17718 consensus_queue.cc:260] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 231, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.555292 17718 raft_consensus.cc:3054] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:57.556042 17718 raft_consensus.cc:2463] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 2.
I20250110 03:06:57.556291 17522 leader_election.cc:304] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:57.556409 18522 raft_consensus.cc:2798] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:57.557178 18522 raft_consensus.cc:695] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:57.557247 18522 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 231, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.558024 17314 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 1 to 2, leader changed from 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:57.559278 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet e365416b475a4288b17d235deddf139d with delete_type TABLET_DATA_TOMBSTONED (TS 92bcfebb99cc47bba372938b64a40f4b not found in new config with opid_index 268) from {username='slave'} at 127.0.0.1:60766
I20250110 03:06:57.559386 19619 tablet_replica.cc:331] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: stopping tablet replica
I20250110 03:06:57.559440 19619 raft_consensus.cc:2238] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.559476 19619 raft_consensus.cc:2267] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.559806 19619 ts_tablet_manager.cc:1905] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:57.559806 17307 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 213 to 228, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "d1e6d3913b04475f8962dd6496fbea0a" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } }
I20250110 03:06:57.560662 17301 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 85fe921f114943c896a041599c39dbe5 with cas_config_opid_index 213: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.560810 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 85fe921f114943c896a041599c39dbe5 with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 228) from {username='slave'} at 127.0.0.1:57906
I20250110 03:06:57.560899 19575 tablet_replica.cc:331] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:06:57.560881 19104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4d68bc455bd4920ba99280f593678ec" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 2 candidate_status { last_received { term: 1 index: 231 } } ignore_live_leader: true dest_uuid: "d28dc7130e374d8486cc68018c03fd34"
I20250110 03:06:57.560949 19104 raft_consensus.cc:3054] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:57.560956 19575 raft_consensus.cc:2238] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.560987 19575 raft_consensus.cc:2267] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.561177 19575 ts_tablet_manager.cc:1905] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250110 03:06:57.574291 17677 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60070: Illegal state: replica 0b345888171f4abcb611a51e27cbcd59 is not leader of this config: current role FOLLOWER
I20250110 03:06:57.619380 17716 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20250110 03:06:57.619783 18522 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20250110 03:06:57.624387 18437 raft_consensus.cc:1059] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b: attempting to promote NON_VOTER b61b88548de84191b17de06cca3c435a to VOTER
I20250110 03:06:57.624526 18437 consensus_queue.cc:237] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:57.625257 17588 raft_consensus.cc:1270] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20250110 03:06:57.625559 19318 consensus_queue.cc:1035] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20250110 03:06:57.625653 18704 raft_consensus.cc:1270] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 LEARNER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch)
I20250110 03:06:57.626138 19171 consensus_queue.cc:1035] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20250110 03:06:57.626880 19318 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index 91 to 216, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.627406 17588 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 91 to 216, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.628803 18704 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index 91 to 216, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.652863 19104 raft_consensus.cc:2463] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 2.
I20250110 03:06:57.653095 19103 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 2 index: 233. (index mismatch)
I20250110 03:06:57.653369 19619 ts_tablet_manager.cc:1918] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.265
I20250110 03:06:57.653435 19619 log.cc:1198] T e365416b475a4288b17d235deddf139d P 92bcfebb99cc47bba372938b64a40f4b: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/e365416b475a4288b17d235deddf139d
I20250110 03:06:57.654224 18523 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.000s
I20250110 03:06:57.657115 17299 catalog_manager.cc:4872] TS 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521): tablet e365416b475a4288b17d235deddf139d (table rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]) successfully deleted
I20250110 03:06:57.657112 17307 catalog_manager.cc:5526] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 91 to 216, b61b88548de84191b17de06cca3c435a (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.658953 17456 consensus_queue.cc:237] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 217, Committed index: 217, Last appended: 1.217, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } }
I20250110 03:06:57.659375 18706 raft_consensus.cc:1270] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 217. Preceding OpId from leader: term: 1 index: 218. (index mismatch)
I20250110 03:06:57.660460 19212 consensus_queue.cc:1035] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 218, Last known committed idx: 217, Time since last communication: 0.000s
I20250110 03:06:57.661630 19212 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.218: config changed from index 216 to 218, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.662412 18706 raft_consensus.cc:2949] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.218: config changed from index 216 to 218, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.756305 19575 ts_tablet_manager.cc:1918] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.224
I20250110 03:06:57.756402 19575 log.cc:1198] T 85fe921f114943c896a041599c39dbe5 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/85fe921f114943c896a041599c39dbe5
I20250110 03:06:57.756803 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet 85fe921f114943c896a041599c39dbe5 (table rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]) successfully deleted
I20250110 03:06:57.756994 17299 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f3ec6ddcb58d48d89f4cea0b3db80035 with cas_config_opid_index 216: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.756969 17307 catalog_manager.cc:5526] T f3ec6ddcb58d48d89f4cea0b3db80035 P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 216 to 218, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 218 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } }
I20250110 03:06:57.758415 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ec6ddcb58d48d89f4cea0b3db80035 with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 218) from {username='slave'} at 127.0.0.1:57906
I20250110 03:06:57.758517 19575 tablet_replica.cc:331] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:06:57.758795 19575 raft_consensus.cc:2238] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.758875 19575 raft_consensus.cc:2267] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.759203 19575 ts_tablet_manager.cc:1905] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:57.758469 17588 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 2.237, Last appended by leader: 231, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:57.760584 19103 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20250110 03:06:57.761376 18523 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20250110 03:06:57.761950 18523 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.238: config changed from index 222 to 238, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.762673 19103 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.238: config changed from index 222 to 238, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.856671 17307 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 reported cstate change: config changed from index 222 to 238, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:57.857540 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d68bc455bd4920ba99280f593678ec with delete_type TABLET_DATA_TOMBSTONED (TS 0b345888171f4abcb611a51e27cbcd59 not found in new config with opid_index 238) from {username='slave'} at 127.0.0.1:40906
I20250110 03:06:57.857746 19970 tablet_replica.cc:331] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: stopping tablet replica
I20250110 03:06:57.857822 19970 raft_consensus.cc:2238] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:57.857893 19970 raft_consensus.cc:2267] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:57.858773 19970 ts_tablet_manager.cc:1905] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:57.860244 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c4d68bc455bd4920ba99280f593678ec with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:57.902680 17697 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:06:57.905691 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:06:57.908712 19575 ts_tablet_manager.cc:1918] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.217
I20250110 03:06:57.908775 19575 log.cc:1198] T f3ec6ddcb58d48d89f4cea0b3db80035 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/f3ec6ddcb58d48d89f4cea0b3db80035
I20250110 03:06:57.909191 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet f3ec6ddcb58d48d89f4cea0b3db80035 (table rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]) successfully deleted
I20250110 03:06:57.911048 19970 ts_tablet_manager.cc:1918] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.237
I20250110 03:06:57.911108 19970 log.cc:1198] T c4d68bc455bd4920ba99280f593678ec P 0b345888171f4abcb611a51e27cbcd59: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/c4d68bc455bd4920ba99280f593678ec
I20250110 03:06:57.911479 17300 catalog_manager.cc:4872] TS 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605): tablet c4d68bc455bd4920ba99280f593678ec (table rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]) successfully deleted
I20250110 03:06:57.913913 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:57.914040 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:57.914275 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:06:57.915903 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:57.925592 19084 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:57.947171 18837 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 273, Committed index: 273, Last appended: 2.274, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:57.947732 17716 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 273. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20250110 03:06:57.948236 19918 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 273, Time since last communication: 0.000s
I20250110 03:06:57.949359 19918 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 228 to 275, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:57.950541 17716 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 228 to 275, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
W20250110 03:06:57.953470 18773 consensus_peers.cc:487] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a -> Peer d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779): Couldn't send request to peer d28dc7130e374d8486cc68018c03fd34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85fe921f114943c896a041599c39dbe5. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:57.955510 17306 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a reported cstate change: config changed from index 228 to 275, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New cstate: current_term: 2 leader_uuid: "d1e6d3913b04475f8962dd6496fbea0a" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:57.967144 17718 consensus_queue.cc:237] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 1.366, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:57.967607 17588 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 366. Preceding OpId from leader: term: 1 index: 367. (index mismatch)
I20250110 03:06:57.967784 19215 consensus_queue.cc:1035] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
W20250110 03:06:57.968024 17652 consensus_peers.cc:487] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 -> Peer 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693): Couldn't send request to peer 4ce3080cc7de4c729423b95fb7158a07. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 701c561086ce457e883794c11cb34ab6. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:57.969425 18569 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.367: config changed from index -1 to 367, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } } }
I20250110 03:06:57.970283 17588 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.367: config changed from index -1 to 367, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } } }
I20250110 03:06:58.058797 17314 catalog_manager.cc:5526] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index -1 to 367, NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:58.069136 17588 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 1.206, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:58.069821 18836 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 206. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
W20250110 03:06:58.069826 17524 consensus_peers.cc:487] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e -> Peer d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779): Couldn't send request to peer d28dc7130e374d8486cc68018c03fd34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7906c51301244bd4a592d8e01e6ad602. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:06:58.070022 18522 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250110 03:06:58.071144 19222 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.207: config changed from index 123 to 207, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:58.071745 18836 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.207: config changed from index 123 to 207, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:58.157512 17314 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a reported cstate change: config changed from index 123 to 207, NON_VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } } }
I20250110 03:06:58.347121 20045 ts_tablet_manager.cc:927] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: Initiating tablet copy from peer 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:06:58.347383 20045 tablet_copy_client.cc:323] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:46605
I20250110 03:06:58.347658 17739 tablet_copy_service.cc:140] P 0b345888171f4abcb611a51e27cbcd59: Received BeginTabletCopySession request for tablet 701c561086ce457e883794c11cb34ab6 from peer 4ce3080cc7de4c729423b95fb7158a07 ({username='slave'} at 127.14.19.134:52927)
I20250110 03:06:58.347728 17739 tablet_copy_service.cc:161] P 0b345888171f4abcb611a51e27cbcd59: Beginning new tablet copy session on tablet 701c561086ce457e883794c11cb34ab6 from peer 4ce3080cc7de4c729423b95fb7158a07 at {username='slave'} at 127.14.19.134:52927: session id = 4ce3080cc7de4c729423b95fb7158a07-701c561086ce457e883794c11cb34ab6
I20250110 03:06:58.348163 17739 tablet_copy_source_session.cc:215] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:58.348686 20045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 701c561086ce457e883794c11cb34ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:58.350395 20045 tablet_copy_client.cc:806] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:58.350513 20045 tablet_copy_client.cc:670] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:58.352064 20045 tablet_copy_client.cc:538] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:58.355376 20045 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap starting.
I20250110 03:06:58.403643 20045 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=425 overwritten=0 applied=425 ignored=0} inserts{seen=1056 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:58.403956 20045 tablet_bootstrap.cc:492] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: Bootstrap complete.
I20250110 03:06:58.404115 20045 ts_tablet_manager.cc:1397] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: Time spent bootstrapping tablet: real 0.049s user 0.034s sys 0.010s
I20250110 03:06:58.404271 20045 raft_consensus.cc:357] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:58.404340 20045 raft_consensus.cc:738] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ce3080cc7de4c729423b95fb7158a07, State: Initialized, Role: LEARNER
I20250110 03:06:58.404361 20045 consensus_queue.cc:260] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 425, Last appended: 1.425, Last appended by leader: 425, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: true } }
I20250110 03:06:58.404438 20045 ts_tablet_manager.cc:1428] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:06:58.404965 17739 tablet_copy_service.cc:342] P 0b345888171f4abcb611a51e27cbcd59: Request end of tablet copy session 4ce3080cc7de4c729423b95fb7158a07-701c561086ce457e883794c11cb34ab6 received from {username='slave'} at 127.14.19.134:52927
I20250110 03:06:58.405025 17739 tablet_copy_service.cc:434] P 0b345888171f4abcb611a51e27cbcd59: ending tablet copy session 4ce3080cc7de4c729423b95fb7158a07-701c561086ce457e883794c11cb34ab6 on tablet 701c561086ce457e883794c11cb34ab6 with peer 4ce3080cc7de4c729423b95fb7158a07
I20250110 03:06:58.462597 20063 ts_tablet_manager.cc:927] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: Initiating tablet copy from peer d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385)
I20250110 03:06:58.462838 20063 tablet_copy_client.cc:323] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.133:39385
I20250110 03:06:58.466280 18857 tablet_copy_service.cc:140] P d1e6d3913b04475f8962dd6496fbea0a: Received BeginTabletCopySession request for tablet 85fe921f114943c896a041599c39dbe5 from peer d28dc7130e374d8486cc68018c03fd34 ({username='slave'} at 127.14.19.135:54941)
I20250110 03:06:58.466343 18857 tablet_copy_service.cc:161] P d1e6d3913b04475f8962dd6496fbea0a: Beginning new tablet copy session on tablet 85fe921f114943c896a041599c39dbe5 from peer d28dc7130e374d8486cc68018c03fd34 at {username='slave'} at 127.14.19.135:54941: session id = d28dc7130e374d8486cc68018c03fd34-85fe921f114943c896a041599c39dbe5
I20250110 03:06:58.466920 18857 tablet_copy_source_session.cc:215] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:58.467329 20063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe921f114943c896a041599c39dbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:58.468760 20063 tablet_copy_client.cc:806] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:58.468866 20063 tablet_copy_client.cc:670] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:58.471004 20063 tablet_copy_client.cc:538] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:58.559038 20063 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: Bootstrap starting.
I20250110 03:06:58.601644 20063 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=344 overwritten=0 applied=343 ignored=0} inserts{seen=970 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 03:06:58.601944 20063 tablet_bootstrap.cc:492] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: Bootstrap complete.
I20250110 03:06:58.602092 20063 ts_tablet_manager.cc:1397] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: Time spent bootstrapping tablet: real 0.043s user 0.014s sys 0.023s
I20250110 03:06:58.602217 20063 raft_consensus.cc:357] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:58.602345 20063 raft_consensus.cc:738] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d28dc7130e374d8486cc68018c03fd34, State: Initialized, Role: LEARNER
I20250110 03:06:58.602396 20063 consensus_queue.cc:260] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 343, Last appended: 2.344, Last appended by leader: 344, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:58.602607 20063 ts_tablet_manager.cc:1428] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20250110 03:06:58.602882 18857 tablet_copy_service.cc:342] P d1e6d3913b04475f8962dd6496fbea0a: Request end of tablet copy session d28dc7130e374d8486cc68018c03fd34-85fe921f114943c896a041599c39dbe5 received from {username='slave'} at 127.14.19.135:54941
I20250110 03:06:58.602960 18857 tablet_copy_service.cc:434] P d1e6d3913b04475f8962dd6496fbea0a: ending tablet copy session d28dc7130e374d8486cc68018c03fd34-85fe921f114943c896a041599c39dbe5 on tablet 85fe921f114943c896a041599c39dbe5 with peer d28dc7130e374d8486cc68018c03fd34
I20250110 03:06:58.622671 20063 ts_tablet_manager.cc:927] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:06:58.622813 20063 tablet_copy_client.cc:323] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:06:58.625630 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet 7906c51301244bd4a592d8e01e6ad602 from peer d28dc7130e374d8486cc68018c03fd34 ({username='slave'} at 127.14.19.135:38583)
I20250110 03:06:58.625702 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet 7906c51301244bd4a592d8e01e6ad602 from peer d28dc7130e374d8486cc68018c03fd34 at {username='slave'} at 127.14.19.135:38583: session id = d28dc7130e374d8486cc68018c03fd34-7906c51301244bd4a592d8e01e6ad602
I20250110 03:06:58.626271 17608 tablet_copy_source_session.cc:215] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:06:58.626616 20063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7906c51301244bd4a592d8e01e6ad602. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:06:58.627836 20063 tablet_copy_client.cc:806] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 0 data blocks...
I20250110 03:06:58.627940 20063 tablet_copy_client.cc:670] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Starting download of 1 WAL segments...
I20250110 03:06:58.629594 20063 tablet_copy_client.cc:538] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:06:58.659044 20063 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: Bootstrap starting.
I20250110 03:06:58.688159 20063 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: Bootstrap replayed 1/1 log segments. Stats: ops{read=261 overwritten=0 applied=261 ignored=0} inserts{seen=889 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:06:58.688402 20063 tablet_bootstrap.cc:492] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: Bootstrap complete.
I20250110 03:06:58.688557 20063 ts_tablet_manager.cc:1397] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.016s
I20250110 03:06:58.688714 20063 raft_consensus.cc:357] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:58.688817 20063 raft_consensus.cc:738] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d28dc7130e374d8486cc68018c03fd34, State: Initialized, Role: LEARNER
I20250110 03:06:58.688855 20063 consensus_queue.cc:260] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 261, Last appended: 1.261, Last appended by leader: 261, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: true } }
I20250110 03:06:58.689002 20063 ts_tablet_manager.cc:1428] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:06:58.689363 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session d28dc7130e374d8486cc68018c03fd34-7906c51301244bd4a592d8e01e6ad602 received from {username='slave'} at 127.14.19.135:38583
I20250110 03:06:58.689427 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session d28dc7130e374d8486cc68018c03fd34-7906c51301244bd4a592d8e01e6ad602 on tablet 7906c51301244bd4a592d8e01e6ad602 with peer d28dc7130e374d8486cc68018c03fd34
I20250110 03:06:58.809315 18968 raft_consensus.cc:1212] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.366->[1.367-1.550] Dedup: 1.425->[1.426-1.550]
I20250110 03:06:58.814756 20020 raft_consensus.cc:1059] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: attempting to promote NON_VOTER 4ce3080cc7de4c729423b95fb7158a07 to VOTER
I20250110 03:06:58.814929 20020 consensus_queue.cc:237] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 551, Committed index: 551, Last appended: 1.551, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:58.815402 18968 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 LEARNER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 551. Preceding OpId from leader: term: 1 index: 552. (index mismatch)
I20250110 03:06:58.815475 17588 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 1 index: 551. Preceding OpId from leader: term: 1 index: 552. (index mismatch)
I20250110 03:06:58.815577 20085 consensus_queue.cc:1035] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 552, Last known committed idx: 551, Time since last communication: 0.000s
I20250110 03:06:58.815804 18569 consensus_queue.cc:1035] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 552, Last known committed idx: 551, Time since last communication: 0.000s
I20250110 03:06:58.816150 18569 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Committing config change with OpId 1.552: config changed from index 367 to 552, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:58.816289 18968 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Committing config change with OpId 1.552: config changed from index 367 to 552, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:58.816438 17588 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Committing config change with OpId 1.552: config changed from index 367 to 552, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:58.862123 19103 raft_consensus.cc:1212] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.274->[2.275-2.344] Dedup: 2.344->[]
I20250110 03:06:58.865000 17306 catalog_manager.cc:5526] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 reported cstate change: config changed from index 367 to 552, 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:58.868237 19901 raft_consensus.cc:1059] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a: attempting to promote NON_VOTER d28dc7130e374d8486cc68018c03fd34 to VOTER
I20250110 03:06:58.868410 19901 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 344, Committed index: 344, Last appended: 2.344, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:58.869129 17718 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20250110 03:06:58.869333 20105 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20250110 03:06:58.869797 19103 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 LEARNER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 344. Preceding OpId from leader: term: 2 index: 345. (index mismatch)
I20250110 03:06:58.870280 19901 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEADER]: Committing config change with OpId 2.345: config changed from index 275 to 345, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:58.870534 20105 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 345, Last known committed idx: 344, Time since last communication: 0.000s
I20250110 03:06:58.870865 17718 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 275 to 345, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:58.871122 19103 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.345: config changed from index 275 to 345, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:58.873029 17716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701c561086ce457e883794c11cb34ab6"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37524
I20250110 03:06:58.873098 17716 raft_consensus.cc:604] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:58.885587 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.888168 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.890615 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.894088 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.902608 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.908020 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.918545 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.929226 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.940894 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.950460 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:58.957302 17311 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a reported cstate change: config changed from index 275 to 345, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d1e6d3913b04475f8962dd6496fbea0a" committed_config { opid_index: 345 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:58.959681 18837 consensus_queue.cc:237] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 2.347, Last appended by leader: 224, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:58.960244 19103 raft_consensus.cc:1270] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer d1e6d3913b04475f8962dd6496fbea0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 347. Preceding OpId from leader: term: 2 index: 348. (index mismatch)
I20250110 03:06:58.960404 19901 consensus_queue.cc:1035] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20250110 03:06:58.960901 20105 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d1e6d3913b04475f8962dd6496fbea0a [term 2 LEADER]: Committing config change with OpId 2.348: config changed from index 345 to 348, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
W20250110 03:06:58.961056 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:58.962318 19103 raft_consensus.cc:2949] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.348: config changed from index 345 to 348, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
W20250110 03:06:58.973501 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:58.988938 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.003455 17678 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.018882 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.034336 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.053859 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.057500 17311 catalog_manager.cc:5526] T 85fe921f114943c896a041599c39dbe5 P d28dc7130e374d8486cc68018c03fd34 reported cstate change: config changed from index 345 to 348, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "d1e6d3913b04475f8962dd6496fbea0a" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:59.058061 17301 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 85fe921f114943c896a041599c39dbe5 with cas_config_opid_index 345: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:59.062131 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet 85fe921f114943c896a041599c39dbe5 with delete_type TABLET_DATA_TOMBSTONED (TS 0b345888171f4abcb611a51e27cbcd59 not found in new config with opid_index 348) from {username='slave'} at 127.0.0.1:40906
I20250110 03:06:59.062297 20124 tablet_replica.cc:331] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: stopping tablet replica
I20250110 03:06:59.062427 20124 raft_consensus.cc:2238] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:59.062496 20124 raft_consensus.cc:2267] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:59.062909 20124 ts_tablet_manager.cc:1905] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250110 03:06:59.075263 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.093953 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.114367 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.114503 19103 raft_consensus.cc:1212] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.206->[1.207-1.278] Dedup: 1.261->[1.262-1.278]
I20250110 03:06:59.116995 18523 raft_consensus.cc:1059] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER d28dc7130e374d8486cc68018c03fd34 to VOTER
I20250110 03:06:59.117144 18523 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:59.117509 18837 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20250110 03:06:59.117527 19103 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20250110 03:06:59.117728 19219 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20250110 03:06:59.117832 20044 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20250110 03:06:59.118278 20044 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 207 to 280, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:59.118379 18837 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 207 to 280, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:59.118731 19103 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 207 to 280, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
W20250110 03:06:59.137969 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.161074 17314 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 207 to 280, d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:59.163683 20124 ts_tablet_manager.cc:1918] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.347
W20250110 03:06:59.163738 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.163764 20124 log.cc:1198] T 85fe921f114943c896a041599c39dbe5 P 0b345888171f4abcb611a51e27cbcd59: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/85fe921f114943c896a041599c39dbe5
I20250110 03:06:59.164203 17300 catalog_manager.cc:4872] TS 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605): tablet 85fe921f114943c896a041599c39dbe5 (table rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]) successfully deleted
I20250110 03:06:59.167904 17588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52446
I20250110 03:06:59.167979 17588 raft_consensus.cc:604] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:59.168226 18523 raft_consensus.cc:988] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: : Instructing follower d28dc7130e374d8486cc68018c03fd34 to start an election
I20250110 03:06:59.168294 19219 raft_consensus.cc:1076] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Signalling peer d28dc7130e374d8486cc68018c03fd34 to start an election
I20250110 03:06:59.168597 19102 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602"
dest_uuid: "d28dc7130e374d8486cc68018c03fd34"
from {username='slave'} at 127.14.19.130:57059
I20250110 03:06:59.168691 19102 raft_consensus.cc:491] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:06:59.168726 19102 raft_consensus.cc:3054] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:59.169888 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
I20250110 03:06:59.174346 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701c561086ce457e883794c11cb34ab6"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37550
I20250110 03:06:59.174407 17718 raft_consensus.cc:604] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:59.174726 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.178149 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
I20250110 03:06:59.183043 17588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52460
I20250110 03:06:59.183096 17588 raft_consensus.cc:604] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:59.184546 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.188581 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.189342 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701c561086ce457e883794c11cb34ab6"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37568
I20250110 03:06:59.189399 17718 raft_consensus.cc:604] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
W20250110 03:06:59.189452 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.196797 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.203161 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.212528 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.214134 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.222031 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.233398 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
W20250110 03:06:59.241741 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
W20250110 03:06:59.246892 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Service unavailable: leader transfer in progress
I20250110 03:06:59.258522 19102 raft_consensus.cc:513] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:59.258770 19103 raft_consensus.cc:1235] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Rejecting Update request from peer 74fcf6c6434b49d8a777aaeda45d971e for earlier term 1. Current term is 2. Ops: []
I20250110 03:06:59.258917 19102 leader_election.cc:290] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [CANDIDATE]: Term 2 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185), d1e6d3913b04475f8962dd6496fbea0a (127.14.19.133:39385)
I20250110 03:06:59.259083 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602" candidate_uuid: "d28dc7130e374d8486cc68018c03fd34" candidate_term: 2 candidate_status { last_received { term: 1 index: 282 } } ignore_live_leader: true dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:06:59.259158 17588 raft_consensus.cc:3049] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:06:59.259186 17588 raft_consensus.cc:738] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:59.259183 18837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7906c51301244bd4a592d8e01e6ad602" candidate_uuid: "d28dc7130e374d8486cc68018c03fd34" candidate_term: 2 candidate_status { last_received { term: 1 index: 282 } } ignore_live_leader: true dest_uuid: "d1e6d3913b04475f8962dd6496fbea0a"
I20250110 03:06:59.259243 17588 consensus_queue.cc:260] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 282, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:59.259280 18837 raft_consensus.cc:3054] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:59.259315 17588 raft_consensus.cc:3054] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:59.259891 17588 raft_consensus.cc:2463] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d28dc7130e374d8486cc68018c03fd34 in term 2.
I20250110 03:06:59.260047 19038 leader_election.cc:304] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74fcf6c6434b49d8a777aaeda45d971e, d28dc7130e374d8486cc68018c03fd34; no voters:
I20250110 03:06:59.260109 18837 raft_consensus.cc:2463] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d28dc7130e374d8486cc68018c03fd34 in term 2.
I20250110 03:06:59.260227 20067 raft_consensus.cc:2798] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:59.260353 20067 raft_consensus.cc:695] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 2 LEADER]: Becoming Leader. State: Replica: d28dc7130e374d8486cc68018c03fd34, State: Running, Role: LEADER
W20250110 03:06:59.260393 17546 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57954: Illegal state: replica 74fcf6c6434b49d8a777aaeda45d971e is not leader of this config: current role FOLLOWER
I20250110 03:06:59.260442 20067 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.282, Last appended by leader: 282, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:59.261049 17311 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 reported cstate change: term changed from 1 to 2, leader changed from 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) to d28dc7130e374d8486cc68018c03fd34 (127.14.19.135). New cstate: current_term: 2 leader_uuid: "d28dc7130e374d8486cc68018c03fd34" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250110 03:06:59.270177 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.294986 17588 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Refusing update from remote peer d28dc7130e374d8486cc68018c03fd34: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20250110 03:06:59.294982 18837 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Refusing update from remote peer d28dc7130e374d8486cc68018c03fd34: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 2 index: 284. (index mismatch)
I20250110 03:06:59.295248 20067 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20250110 03:06:59.295400 20174 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
W20250110 03:06:59.299654 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.300956 19102 consensus_queue.cc:237] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 2.284, Last appended by leader: 282, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } }
I20250110 03:06:59.301359 18837 raft_consensus.cc:1270] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Refusing update from remote peer d28dc7130e374d8486cc68018c03fd34: Log matching property violated. Preceding OpId in replica: term: 2 index: 284. Preceding OpId from leader: term: 2 index: 285. (index mismatch)
I20250110 03:06:59.301548 20178 consensus_queue.cc:1035] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20250110 03:06:59.302065 20067 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d28dc7130e374d8486cc68018c03fd34 [term 2 LEADER]: Committing config change with OpId 2.285: config changed from index 280 to 285, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:59.302176 18837 raft_consensus.cc:2949] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a [term 2 FOLLOWER]: Committing config change with OpId 2.285: config changed from index 280 to 285, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
W20250110 03:06:59.330612 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.353173 17302 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7906c51301244bd4a592d8e01e6ad602 with cas_config_opid_index 280: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:59.353533 17306 catalog_manager.cc:5526] T 7906c51301244bd4a592d8e01e6ad602 P d1e6d3913b04475f8962dd6496fbea0a reported cstate change: config changed from index 280 to 285, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "d28dc7130e374d8486cc68018c03fd34" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "d1e6d3913b04475f8962dd6496fbea0a" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 39385 } attrs { promote: false } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } }
I20250110 03:06:59.354318 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 7906c51301244bd4a592d8e01e6ad602 with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 285) from {username='slave'} at 127.0.0.1:57906
I20250110 03:06:59.354537 20181 tablet_replica.cc:331] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:06:59.354636 20181 raft_consensus.cc:2238] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:59.354694 20181 raft_consensus.cc:2267] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:59.355199 20181 ts_tablet_manager.cc:1905] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250110 03:06:59.363126 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.375967 19215 raft_consensus.cc:988] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: : Instructing follower 74fcf6c6434b49d8a777aaeda45d971e to start an election
I20250110 03:06:59.376030 19215 raft_consensus.cc:1076] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Signalling peer 74fcf6c6434b49d8a777aaeda45d971e to start an election
I20250110 03:06:59.376230 17588 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "701c561086ce457e883794c11cb34ab6"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
from {username='slave'} at 127.14.19.131:50761
I20250110 03:06:59.376308 17588 raft_consensus.cc:491] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:06:59.376338 17588 raft_consensus.cc:3054] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:59.397012 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.402163 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701c561086ce457e883794c11cb34ab6"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37596
I20250110 03:06:59.402244 17718 raft_consensus.cc:604] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:59.410144 17718 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "701c561086ce457e883794c11cb34ab6"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37602
I20250110 03:06:59.410225 17718 raft_consensus.cc:604] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:06:59.413776 20181 ts_tablet_manager.cc:1918] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.284
I20250110 03:06:59.413842 20181 log.cc:1198] T 7906c51301244bd4a592d8e01e6ad602 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/7906c51301244bd4a592d8e01e6ad602
I20250110 03:06:59.414256 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet 7906c51301244bd4a592d8e01e6ad602 (table rebalance_test_table_0 [id=0faf368fab8c4df98faa6ad34344b290]) successfully deleted
W20250110 03:06:59.428591 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Service unavailable: leader transfer in progress
I20250110 03:06:59.452842 17588 raft_consensus.cc:513] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:59.452989 17588 leader_election.cc:290] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605), 4ce3080cc7de4c729423b95fb7158a07 (127.14.19.134:36693)
I20250110 03:06:59.453207 18968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701c561086ce457e883794c11cb34ab6" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 2 candidate_status { last_received { term: 1 index: 553 } } ignore_live_leader: true dest_uuid: "4ce3080cc7de4c729423b95fb7158a07"
I20250110 03:06:59.453294 18968 raft_consensus.cc:3054] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:06:59.454114 18968 raft_consensus.cc:2463] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 2.
I20250110 03:06:59.454263 17521 leader_election.cc:304] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ce3080cc7de4c729423b95fb7158a07, 74fcf6c6434b49d8a777aaeda45d971e; no voters:
I20250110 03:06:59.454355 18523 raft_consensus.cc:2798] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:06:59.454411 18523 raft_consensus.cc:695] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Becoming Leader. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:06:59.454455 18523 consensus_queue.cc:237] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 553, Committed index: 553, Last appended: 1.553, Last appended by leader: 553, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:59.456712 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "701c561086ce457e883794c11cb34ab6" candidate_uuid: "74fcf6c6434b49d8a777aaeda45d971e" candidate_term: 2 candidate_status { last_received { term: 1 index: 553 } } ignore_live_leader: true dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:06:59.456792 17718 raft_consensus.cc:3049] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:06:59.456827 17718 raft_consensus.cc:738] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:06:59.456880 17718 consensus_queue.cc:260] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 553, Committed index: 553, Last appended: 1.553, Last appended by leader: 553, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:59.456838 17311 catalog_manager.cc:5526] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: term changed from 1 to 2, leader changed from 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) to 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130). New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 552 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:06:59.456955 17718 raft_consensus.cc:3054] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 1 FOLLOWER]: Advancing to term 2
W20250110 03:06:59.461988 17679 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60066: Illegal state: replica 0b345888171f4abcb611a51e27cbcd59 is not leader of this config: current role FOLLOWER
I20250110 03:06:59.499982 18968 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 553. Preceding OpId from leader: term: 2 index: 555. (index mismatch)
I20250110 03:06:59.500346 18523 consensus_queue.cc:1035] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 554, Last known committed idx: 553, Time since last communication: 0.000s
I20250110 03:06:59.502394 17588 consensus_queue.cc:237] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 555, Committed index: 555, Last appended: 2.555, Last appended by leader: 553, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } }
I20250110 03:06:59.503104 18968 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 555. Preceding OpId from leader: term: 2 index: 556. (index mismatch)
I20250110 03:06:59.503557 18523 consensus_queue.cc:1035] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 556, Last known committed idx: 555, Time since last communication: 0.000s
I20250110 03:06:59.504487 20044 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.556: config changed from index 552 to 556, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:59.504609 18969 raft_consensus.cc:2949] T 701c561086ce457e883794c11cb34ab6 P 4ce3080cc7de4c729423b95fb7158a07 [term 2 FOLLOWER]: Committing config change with OpId 2.556: config changed from index 552 to 556, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } } }
I20250110 03:06:59.563297 17718 raft_consensus.cc:2463] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74fcf6c6434b49d8a777aaeda45d971e in term 2.
I20250110 03:06:59.563479 17716 raft_consensus.cc:1270] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 553. Preceding OpId from leader: term: 2 index: 555. (index mismatch)
I20250110 03:06:59.563706 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 701c561086ce457e883794c11cb34ab6 with cas_config_opid_index 552: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:06:59.563688 17311 catalog_manager.cc:5526] T 701c561086ce457e883794c11cb34ab6 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 552 to 556, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 556 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ce3080cc7de4c729423b95fb7158a07" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 36693 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:06:59.565299 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet 701c561086ce457e883794c11cb34ab6 with delete_type TABLET_DATA_TOMBSTONED (TS 0b345888171f4abcb611a51e27cbcd59 not found in new config with opid_index 556) from {username='slave'} at 127.0.0.1:40906
I20250110 03:06:59.565407 20124 tablet_replica.cc:331] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: stopping tablet replica
I20250110 03:06:59.565455 20124 raft_consensus.cc:2238] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:06:59.565505 20124 raft_consensus.cc:2267] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:06:59.566172 20124 ts_tablet_manager.cc:1905] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:06:59.587743 20124 ts_tablet_manager.cc:1918] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.553
I20250110 03:06:59.587826 20124 log.cc:1198] T 701c561086ce457e883794c11cb34ab6 P 0b345888171f4abcb611a51e27cbcd59: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/701c561086ce457e883794c11cb34ab6
I20250110 03:06:59.588254 17300 catalog_manager.cc:4872] TS 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605): tablet 701c561086ce457e883794c11cb34ab6 (table rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]) successfully deleted
I20250110 03:06:59.661046 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:06:59.665407 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:59.667310 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:06:59.673890 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:06:59.675879 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:59.680336 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:59.680507 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:59.760040 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:06:59.765129 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:06:59.766866 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:06:59.775256 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:06:59.781499 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:06:59.790597 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:06:59.791057 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.005041 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:00.005632 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:00.011466 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.011570 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:00.016768 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:00.017336 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.027515 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:00.094825 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:00.101960 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:00.104975 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.109829 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:00.111640 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.114355 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:00.114602 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:00.920712 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:00.921128 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:00.924703 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.925022 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:00.927884 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:00.928400 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:00.931995 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c800037a95e4de6bd43d671ea55076b | 127.14.19.190:35275 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:42975 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0b345888171f4abcb611a51e27cbcd59 | 127.14.19.131:46605 | HEALTHY | <none> | 3 | 0
4ce3080cc7de4c729423b95fb7158a07 | 127.14.19.134:36693 | HEALTHY | <none> | 2 | 0
74fcf6c6434b49d8a777aaeda45d971e | 127.14.19.130:35185 | HEALTHY | <none> | 5 | 0
92bcfebb99cc47bba372938b64a40f4b | 127.14.19.129:42521 | HEALTHY | <none> | 4 | 0
b61b88548de84191b17de06cca3c435a | 127.14.19.132:38659 | HEALTHY | <none> | 0 | 0
d1e6d3913b04475f8962dd6496fbea0a | 127.14.19.133:39385 | HEALTHY | <none> | 1 | 0
d28dc7130e374d8486cc68018c03fd34 | 127.14.19.135:32779 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.14.19.129 | experimental | 127.14.19.129:42521
local_ip_for_outbound_sockets | 127.14.19.130 | experimental | 127.14.19.130:35185
local_ip_for_outbound_sockets | 127.14.19.131 | experimental | 127.14.19.131:46605
local_ip_for_outbound_sockets | 127.14.19.132 | experimental | 127.14.19.132:38659
local_ip_for_outbound_sockets | 127.14.19.133 | experimental | 127.14.19.133:39385
local_ip_for_outbound_sockets | 127.14.19.134 | experimental | 127.14.19.134:36693
local_ip_for_outbound_sockets | 127.14.19.135 | experimental | 127.14.19.135:32779
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.19.129:42521
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.19.130:35185
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.19.131:46605
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.19.132:38659
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.19.133:39385
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.19.134:36693
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.19.135:32779
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:42975 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 03:07:00.943173 14414 log_verifier.cc:126] Checking tablet 10fb55ba712c42ee95fcf9f300a54ea0
I20250110 03:07:00.972288 14414 log_verifier.cc:177] Verified matching terms for 439 ops in tablet 10fb55ba712c42ee95fcf9f300a54ea0
I20250110 03:07:00.972514 14414 log_verifier.cc:126] Checking tablet 567885520d33460fae980ae73fabcd6a
I20250110 03:07:00.999780 14414 log_verifier.cc:177] Verified matching terms for 437 ops in tablet 567885520d33460fae980ae73fabcd6a
I20250110 03:07:00.999847 14414 log_verifier.cc:126] Checking tablet 593fb2f6a0a7417ca54b800249318eae
I20250110 03:07:01.029218 14414 log_verifier.cc:177] Verified matching terms for 516 ops in tablet 593fb2f6a0a7417ca54b800249318eae
I20250110 03:07:01.029287 14414 log_verifier.cc:126] Checking tablet 701c561086ce457e883794c11cb34ab6
I20250110 03:07:01.063658 14414 log_verifier.cc:177] Verified matching terms for 680 ops in tablet 701c561086ce457e883794c11cb34ab6
I20250110 03:07:01.063767 14414 log_verifier.cc:126] Checking tablet 7906c51301244bd4a592d8e01e6ad602
I20250110 03:07:01.093111 14414 log_verifier.cc:177] Verified matching terms for 421 ops in tablet 7906c51301244bd4a592d8e01e6ad602
I20250110 03:07:01.093195 14414 log_verifier.cc:126] Checking tablet 83d2ceb09bc5442f8c329ca6b3ad6f35
I20250110 03:07:01.121179 14414 log_verifier.cc:177] Verified matching terms for 414 ops in tablet 83d2ceb09bc5442f8c329ca6b3ad6f35
I20250110 03:07:01.121249 14414 log_verifier.cc:126] Checking tablet 85fe921f114943c896a041599c39dbe5
I20250110 03:07:01.151461 14414 log_verifier.cc:177] Verified matching terms for 467 ops in tablet 85fe921f114943c896a041599c39dbe5
I20250110 03:07:01.151547 14414 log_verifier.cc:126] Checking tablet 888f2d12f3254197a6a490e23016bd46
I20250110 03:07:01.162815 14414 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 888f2d12f3254197a6a490e23016bd46
I20250110 03:07:01.162869 14414 log_verifier.cc:126] Checking tablet 8b2221475806475c83aa9867a548f51f
I20250110 03:07:01.192812 14414 log_verifier.cc:177] Verified matching terms for 484 ops in tablet 8b2221475806475c83aa9867a548f51f
I20250110 03:07:01.192883 14414 log_verifier.cc:126] Checking tablet 8dec030e45a84eb89321f275d0013826
I20250110 03:07:01.222232 14414 log_verifier.cc:177] Verified matching terms for 529 ops in tablet 8dec030e45a84eb89321f275d0013826
I20250110 03:07:01.222303 14414 log_verifier.cc:126] Checking tablet c4d68bc455bd4920ba99280f593678ec
I20250110 03:07:01.252278 14414 log_verifier.cc:177] Verified matching terms for 515 ops in tablet c4d68bc455bd4920ba99280f593678ec
I20250110 03:07:01.252351 14414 log_verifier.cc:126] Checking tablet c8b6b5410b4e4405b419e22daf379c6a
I20250110 03:07:01.280388 14414 log_verifier.cc:177] Verified matching terms for 462 ops in tablet c8b6b5410b4e4405b419e22daf379c6a
I20250110 03:07:01.280455 14414 log_verifier.cc:126] Checking tablet e365416b475a4288b17d235deddf139d
I20250110 03:07:01.314353 14414 log_verifier.cc:177] Verified matching terms for 648 ops in tablet e365416b475a4288b17d235deddf139d
I20250110 03:07:01.314422 14414 log_verifier.cc:126] Checking tablet f3ec6ddcb58d48d89f4cea0b3db80035
I20250110 03:07:01.343441 14414 log_verifier.cc:177] Verified matching terms for 518 ops in tablet f3ec6ddcb58d48d89f4cea0b3db80035
I20250110 03:07:01.343547 14414 log_verifier.cc:126] Checking tablet f8b5177de1fa4e1c9668453452f2064f
I20250110 03:07:01.380677 14414 log_verifier.cc:177] Verified matching terms for 657 ops in tablet f8b5177de1fa4e1c9668453452f2064f
I20250110 03:07:01.380772 14414 log_verifier.cc:126] Checking tablet fa58142d4f86446b99561d0a0fb9fc96
I20250110 03:07:01.407402 14414 log_verifier.cc:177] Verified matching terms for 426 ops in tablet fa58142d4f86446b99561d0a0fb9fc96
I20250110 03:07:01.408452 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.136:0
--local_ip_for_outbound_sockets=127.14.19.136
--webserver_interface=127.14.19.136
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--builtin_ntp_servers=127.14.19.148:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:01.486210 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:01.486349 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:01.486367 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:01.487676 20418 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:01.487717 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:01.487740 20418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.136
I20250110 03:07:01.489076 20418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.19.136
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35275
--never_fsync=true
--heap_profile_path=/tmp/kudu.20418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.136
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:01.489218 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:01.489382 20418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:01.494177 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:01.495188 20418 server_base.cc:1034] running on GCE node
W20250110 03:07:01.495951 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:01.496804 20426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:01.496979 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:01.497152 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:01.498266 20418 hybrid_clock.cc:648] HybridClock initialized: now 1736478421498257 us; error 24 us; skew 500 ppm
I20250110 03:07:01.504848 20418 webserver.cc:458] Webserver started at http://127.14.19.136:34247/ using document root <none> and password file <none>
I20250110 03:07:01.505075 20418 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:01.505136 20418 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:01.505239 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:01.506351 20418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "674934428b684f929c7b3e408795d6de"
format_stamp: "Formatted at 2025-01-10 03:07:01 on dist-test-slave-nrvn"
I20250110 03:07:01.506748 20418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "674934428b684f929c7b3e408795d6de"
format_stamp: "Formatted at 2025-01-10 03:07:01 on dist-test-slave-nrvn"
I20250110 03:07:01.508396 20418 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:07:01.509219 20432 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:01.509392 20418 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:01.509470 20418 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "674934428b684f929c7b3e408795d6de"
format_stamp: "Formatted at 2025-01-10 03:07:01 on dist-test-slave-nrvn"
I20250110 03:07:01.509527 20418 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:01.524665 20418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:01.524863 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:01.524948 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:01.525087 20418 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:01.525359 20418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:01.525393 20418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:01.525424 20418 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:01.525449 20418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:01.530745 20418 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.136:35499
I20250110 03:07:01.530808 20545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.136:35499 every 8 connection(s)
I20250110 03:07:01.531031 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20250110 03:07:01.533128 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 20418
I20250110 03:07:01.533223 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20250110 03:07:01.539762 20546 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35275
I20250110 03:07:01.539870 20546 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:01.540092 20546 heartbeater.cc:510] Master 127.14.19.190:35275 requested a full tablet report, sending...
I20250110 03:07:01.540452 17311 ts_manager.cc:194] Registered new tserver with Master: 674934428b684f929c7b3e408795d6de (127.14.19.136:35499)
I20250110 03:07:01.540993 17311 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.136:48831
I20250110 03:07:01.640857 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:01.641048 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:01.641994 20480 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:01.647297 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:01.650285 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:01.654616 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:01.654964 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:01.655558 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:01.694280 20480 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:01.695222 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:01.700644 17697 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:01.702029 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:01.704439 17568 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:01.704617 18686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:01.705327 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:01.705693 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:01.722658 17456 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 1.516, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:01.723099 18705 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch)
I20250110 03:07:01.723294 20640 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20250110 03:07:01.723985 20263 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.517: config changed from index 244 to 517, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.725136 17307 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index 244 to 517, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:01.725560 18705 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 244 to 517, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
W20250110 03:07:01.725919 17392 consensus_peers.cc:487] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b -> Peer 674934428b684f929c7b3e408795d6de (127.14.19.136:35499): Couldn't send request to peer 674934428b684f929c7b3e408795d6de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 593fb2f6a0a7417ca54b800249318eae. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:01.729624 17588 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 515, Committed index: 515, Last appended: 2.515, Last appended by leader: 231, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:01.730070 19103 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 515. Preceding OpId from leader: term: 2 index: 516. (index mismatch)
I20250110 03:07:01.730261 18523 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 516, Last known committed idx: 515, Time since last communication: 0.000s
I20250110 03:07:01.730944 20653 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.516: config changed from index 238 to 516, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.731069 19103 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.516: config changed from index 238 to 516, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.732105 17311 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 238 to 516, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:01.733217 17523 consensus_peers.cc:487] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e -> Peer 674934428b684f929c7b3e408795d6de (127.14.19.136:35499): Couldn't send request to peer 674934428b684f929c7b3e408795d6de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4d68bc455bd4920ba99280f593678ec. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:01.735882 17588 consensus_queue.cc:237] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 439, Committed index: 439, Last appended: 1.439, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:01.736235 17456 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 439. Preceding OpId from leader: term: 1 index: 440. (index mismatch)
W20250110 03:07:01.736296 17523 consensus_peers.cc:487] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e -> Peer 674934428b684f929c7b3e408795d6de (127.14.19.136:35499): Couldn't send request to peer 674934428b684f929c7b3e408795d6de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10fb55ba712c42ee95fcf9f300a54ea0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:01.736425 20653 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 440, Last known committed idx: 439, Time since last communication: 0.000s
I20250110 03:07:01.736974 18523 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.440: config changed from index -1 to 440, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.737071 17456 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.440: config changed from index -1 to 440, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.738152 17311 catalog_manager.cc:5526] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index -1 to 440, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.741808 17716 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 648, Committed index: 648, Last appended: 2.648, Last appended by leader: 265, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 649 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:01.742276 19103 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 2 index: 648. Preceding OpId from leader: term: 2 index: 649. (index mismatch)
I20250110 03:07:01.742501 20295 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 649, Last known committed idx: 648, Time since last communication: 0.000s
I20250110 03:07:01.743063 20666 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Committing config change with OpId 2.649: config changed from index 268 to 649, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 649 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.743181 19103 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.649: config changed from index 268 to 649, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New config: { opid_index: 649 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } } }
I20250110 03:07:01.744181 17307 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 reported cstate change: config changed from index 268 to 649, NON_VOTER 674934428b684f929c7b3e408795d6de (127.14.19.136) added. New cstate: current_term: 2 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 649 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:01.745185 17654 consensus_peers.cc:487] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 -> Peer 674934428b684f929c7b3e408795d6de (127.14.19.136:35499): Couldn't send request to peer 674934428b684f929c7b3e408795d6de. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e365416b475a4288b17d235deddf139d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:02.212261 20730 ts_tablet_manager.cc:927] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Initiating tablet copy from peer 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521)
I20250110 03:07:02.212464 20730 tablet_copy_client.cc:323] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42521
I20250110 03:07:02.215035 17477 tablet_copy_service.cc:140] P 92bcfebb99cc47bba372938b64a40f4b: Received BeginTabletCopySession request for tablet 593fb2f6a0a7417ca54b800249318eae from peer 674934428b684f929c7b3e408795d6de ({username='slave'} at 127.14.19.136:42341)
I20250110 03:07:02.215097 17477 tablet_copy_service.cc:161] P 92bcfebb99cc47bba372938b64a40f4b: Beginning new tablet copy session on tablet 593fb2f6a0a7417ca54b800249318eae from peer 674934428b684f929c7b3e408795d6de at {username='slave'} at 127.14.19.136:42341: session id = 674934428b684f929c7b3e408795d6de-593fb2f6a0a7417ca54b800249318eae
I20250110 03:07:02.215569 17477 tablet_copy_source_session.cc:215] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:02.216240 20730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 593fb2f6a0a7417ca54b800249318eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:02.218067 20730 tablet_copy_client.cc:806] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:02.218183 20730 tablet_copy_client.cc:670] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:02.219873 20730 tablet_copy_client.cc:538] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:02.220469 20730 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Bootstrap starting.
I20250110 03:07:02.227389 20730 log.cc:826] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:02.255517 20730 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Bootstrap replayed 1/1 log segments. Stats: ops{read=517 overwritten=0 applied=517 ignored=0} inserts{seen=1145 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:02.255760 20730 tablet_bootstrap.cc:492] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Bootstrap complete.
I20250110 03:07:02.255904 20730 ts_tablet_manager.cc:1397] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.023s
I20250110 03:07:02.256647 20730 raft_consensus.cc:357] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.256767 20730 raft_consensus.cc:738] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 674934428b684f929c7b3e408795d6de, State: Initialized, Role: LEARNER
I20250110 03:07:02.256814 20730 consensus_queue.cc:260] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 517, Last appended: 1.517, Last appended by leader: 517, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.257015 20546 heartbeater.cc:502] Master 127.14.19.190:35275 was elected leader, sending a full tablet report...
I20250110 03:07:02.257046 20730 ts_tablet_manager.cc:1428] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:02.257385 17477 tablet_copy_service.cc:342] P 92bcfebb99cc47bba372938b64a40f4b: Request end of tablet copy session 674934428b684f929c7b3e408795d6de-593fb2f6a0a7417ca54b800249318eae received from {username='slave'} at 127.14.19.136:42341
I20250110 03:07:02.257473 17477 tablet_copy_service.cc:434] P 92bcfebb99cc47bba372938b64a40f4b: ending tablet copy session 674934428b684f929c7b3e408795d6de-593fb2f6a0a7417ca54b800249318eae on tablet 593fb2f6a0a7417ca54b800249318eae with peer 674934428b684f929c7b3e408795d6de
I20250110 03:07:02.276278 20730 ts_tablet_manager.cc:927] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: Initiating tablet copy from peer 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605)
I20250110 03:07:02.276432 20730 tablet_copy_client.cc:323] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:46605
I20250110 03:07:02.278769 17739 tablet_copy_service.cc:140] P 0b345888171f4abcb611a51e27cbcd59: Received BeginTabletCopySession request for tablet e365416b475a4288b17d235deddf139d from peer 674934428b684f929c7b3e408795d6de ({username='slave'} at 127.14.19.136:57293)
I20250110 03:07:02.278841 17739 tablet_copy_service.cc:161] P 0b345888171f4abcb611a51e27cbcd59: Beginning new tablet copy session on tablet e365416b475a4288b17d235deddf139d from peer 674934428b684f929c7b3e408795d6de at {username='slave'} at 127.14.19.136:57293: session id = 674934428b684f929c7b3e408795d6de-e365416b475a4288b17d235deddf139d
I20250110 03:07:02.279171 17739 tablet_copy_source_session.cc:215] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:02.279436 20730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e365416b475a4288b17d235deddf139d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:02.280247 20730 tablet_copy_client.cc:806] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:02.280346 20730 tablet_copy_client.cc:670] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:02.282351 20730 tablet_copy_client.cc:538] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:02.282907 20730 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: Bootstrap starting.
I20250110 03:07:02.290714 20737 ts_tablet_manager.cc:927] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:07:02.290874 20737 tablet_copy_client.cc:323] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:07:02.293162 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet c4d68bc455bd4920ba99280f593678ec from peer 674934428b684f929c7b3e408795d6de ({username='slave'} at 127.14.19.136:55717)
I20250110 03:07:02.293217 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet c4d68bc455bd4920ba99280f593678ec from peer 674934428b684f929c7b3e408795d6de at {username='slave'} at 127.14.19.136:55717: session id = 674934428b684f929c7b3e408795d6de-c4d68bc455bd4920ba99280f593678ec
I20250110 03:07:02.293593 17608 tablet_copy_source_session.cc:215] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:02.293876 20737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4d68bc455bd4920ba99280f593678ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:02.295024 20737 tablet_copy_client.cc:806] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:02.295202 20737 tablet_copy_client.cc:670] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:02.296857 20737 tablet_copy_client.cc:538] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:02.297533 20737 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: Bootstrap starting.
I20250110 03:07:02.337047 20740 ts_tablet_manager.cc:927] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: Initiating tablet copy from peer 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185)
I20250110 03:07:02.337240 20740 tablet_copy_client.cc:323] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35185
I20250110 03:07:02.337417 17608 tablet_copy_service.cc:140] P 74fcf6c6434b49d8a777aaeda45d971e: Received BeginTabletCopySession request for tablet 10fb55ba712c42ee95fcf9f300a54ea0 from peer 674934428b684f929c7b3e408795d6de ({username='slave'} at 127.14.19.136:55717)
I20250110 03:07:02.337491 17608 tablet_copy_service.cc:161] P 74fcf6c6434b49d8a777aaeda45d971e: Beginning new tablet copy session on tablet 10fb55ba712c42ee95fcf9f300a54ea0 from peer 674934428b684f929c7b3e408795d6de at {username='slave'} at 127.14.19.136:55717: session id = 674934428b684f929c7b3e408795d6de-10fb55ba712c42ee95fcf9f300a54ea0
I20250110 03:07:02.337874 17608 tablet_copy_source_session.cc:215] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:02.338176 20740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fb55ba712c42ee95fcf9f300a54ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:02.339094 20740 tablet_copy_client.cc:806] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:02.339246 20740 tablet_copy_client.cc:670] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:02.340418 20740 tablet_copy_client.cc:538] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:02.340939 20740 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: Bootstrap starting.
I20250110 03:07:02.343719 20730 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: Bootstrap replayed 1/1 log segments. Stats: ops{read=649 overwritten=0 applied=649 ignored=0} inserts{seen=1330 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:02.343938 20730 tablet_bootstrap.cc:492] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: Bootstrap complete.
I20250110 03:07:02.344110 20730 ts_tablet_manager.cc:1397] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: Time spent bootstrapping tablet: real 0.061s user 0.038s sys 0.022s
I20250110 03:07:02.344259 20730 raft_consensus.cc:357] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 649 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.344347 20730 raft_consensus.cc:738] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 674934428b684f929c7b3e408795d6de, State: Initialized, Role: LEARNER
I20250110 03:07:02.344437 20730 consensus_queue.cc:260] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 649, Last appended: 2.649, Last appended by leader: 649, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 649 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.344558 20730 ts_tablet_manager.cc:1428] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de: Time spent starting tablet: real 0.000s user 0.001s sys 0.001s
I20250110 03:07:02.344851 17739 tablet_copy_service.cc:342] P 0b345888171f4abcb611a51e27cbcd59: Request end of tablet copy session 674934428b684f929c7b3e408795d6de-e365416b475a4288b17d235deddf139d received from {username='slave'} at 127.14.19.136:57293
I20250110 03:07:02.344931 17739 tablet_copy_service.cc:434] P 0b345888171f4abcb611a51e27cbcd59: ending tablet copy session 674934428b684f929c7b3e408795d6de-e365416b475a4288b17d235deddf139d on tablet e365416b475a4288b17d235deddf139d with peer 674934428b684f929c7b3e408795d6de
I20250110 03:07:02.350720 20737 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: Bootstrap replayed 1/1 log segments. Stats: ops{read=516 overwritten=0 applied=516 ignored=0} inserts{seen=1143 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:02.350965 20737 tablet_bootstrap.cc:492] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: Bootstrap complete.
I20250110 03:07:02.351095 20737 ts_tablet_manager.cc:1397] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: Time spent bootstrapping tablet: real 0.054s user 0.029s sys 0.024s
I20250110 03:07:02.351190 20737 raft_consensus.cc:357] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.351244 20737 raft_consensus.cc:738] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 674934428b684f929c7b3e408795d6de, State: Initialized, Role: LEARNER
I20250110 03:07:02.351275 20737 consensus_queue.cc:260] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 516, Last appended: 2.516, Last appended by leader: 516, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 516 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.351387 20737 ts_tablet_manager.cc:1428] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:02.351581 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session 674934428b684f929c7b3e408795d6de-c4d68bc455bd4920ba99280f593678ec received from {username='slave'} at 127.14.19.136:55717
I20250110 03:07:02.351650 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session 674934428b684f929c7b3e408795d6de-c4d68bc455bd4920ba99280f593678ec on tablet c4d68bc455bd4920ba99280f593678ec with peer 674934428b684f929c7b3e408795d6de
I20250110 03:07:02.369980 20740 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: Bootstrap replayed 1/1 log segments. Stats: ops{read=440 overwritten=0 applied=440 ignored=0} inserts{seen=1112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:02.370211 20740 tablet_bootstrap.cc:492] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: Bootstrap complete.
I20250110 03:07:02.370344 20740 ts_tablet_manager.cc:1397] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.007s
I20250110 03:07:02.370450 20740 raft_consensus.cc:357] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.370507 20740 raft_consensus.cc:738] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 674934428b684f929c7b3e408795d6de, State: Initialized, Role: LEARNER
I20250110 03:07:02.370584 20740 consensus_queue.cc:260] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 440, Last appended: 1.440, Last appended by leader: 440, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 440 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: true } }
I20250110 03:07:02.370687 20740 ts_tablet_manager.cc:1428] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de: Time spent starting tablet: real 0.000s user 0.004s sys 0.000s
I20250110 03:07:02.370905 17608 tablet_copy_service.cc:342] P 74fcf6c6434b49d8a777aaeda45d971e: Request end of tablet copy session 674934428b684f929c7b3e408795d6de-10fb55ba712c42ee95fcf9f300a54ea0 received from {username='slave'} at 127.14.19.136:55717
I20250110 03:07:02.370958 17608 tablet_copy_service.cc:434] P 74fcf6c6434b49d8a777aaeda45d971e: ending tablet copy session 674934428b684f929c7b3e408795d6de-10fb55ba712c42ee95fcf9f300a54ea0 on tablet 10fb55ba712c42ee95fcf9f300a54ea0 with peer 674934428b684f929c7b3e408795d6de
I20250110 03:07:02.644462 20500 raft_consensus.cc:1212] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.516->[1.517-1.517] Dedup: 1.517->[]
I20250110 03:07:02.802129 20500 raft_consensus.cc:1212] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.439->[1.440-1.440] Dedup: 1.440->[]
I20250110 03:07:02.826910 20500 raft_consensus.cc:1212] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Deduplicated request from leader. Original: 2.515->[2.516-2.516] Dedup: 2.516->[]
I20250110 03:07:02.895498 20500 raft_consensus.cc:1212] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Deduplicated request from leader. Original: 2.648->[2.649-2.649] Dedup: 2.649->[]
I20250110 03:07:03.195775 20784 raft_consensus.cc:1059] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER 674934428b684f929c7b3e408795d6de to VOTER
I20250110 03:07:03.195924 20784 consensus_queue.cc:237] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 440, Committed index: 440, Last appended: 1.440, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.196373 20500 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20250110 03:07:03.196388 17456 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 1 index: 440. Preceding OpId from leader: term: 1 index: 441. (index mismatch)
I20250110 03:07:03.196578 20660 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20250110 03:07:03.196691 20784 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 441, Last known committed idx: 440, Time since last communication: 0.000s
I20250110 03:07:03.197328 20784 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Committing config change with OpId 1.441: config changed from index 440 to 441, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.197458 17456 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 440 to 441, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.197842 20500 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 1 FOLLOWER]: Committing config change with OpId 1.441: config changed from index 440 to 441, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.198784 17307 catalog_manager.cc:5526] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 440 to 441, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:03.245676 17588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52712
I20250110 03:07:03.245755 17588 raft_consensus.cc:604] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:03.254317 20861 raft_consensus.cc:1059] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: attempting to promote NON_VOTER 674934428b684f929c7b3e408795d6de to VOTER
I20250110 03:07:03.254472 20861 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 517, Committed index: 517, Last appended: 1.517, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.254952 20500 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 1 LEARNER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 517. Preceding OpId from leader: term: 1 index: 518. (index mismatch)
I20250110 03:07:03.254978 18705 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 517. Preceding OpId from leader: term: 1 index: 518. (index mismatch)
I20250110 03:07:03.255123 20863 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 518, Last known committed idx: 517, Time since last communication: 0.000s
I20250110 03:07:03.255235 20861 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 518, Last known committed idx: 517, Time since last communication: 0.000s
I20250110 03:07:03.255956 20863 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Committing config change with OpId 1.518: config changed from index 517 to 518, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.256091 18705 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Committing config change with OpId 1.518: config changed from index 517 to 518, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.256094 20500 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 1 FOLLOWER]: Committing config change with OpId 1.518: config changed from index 517 to 518, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.257185 17311 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index 517 to 518, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:03.358052 20668 raft_consensus.cc:1059] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: attempting to promote NON_VOTER 674934428b684f929c7b3e408795d6de to VOTER
I20250110 03:07:03.358242 20668 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 649, Committed index: 649, Last appended: 2.649, Last appended by leader: 265, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.358733 20500 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 2 index: 649. Preceding OpId from leader: term: 2 index: 650. (index mismatch)
I20250110 03:07:03.358814 19103 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 0b345888171f4abcb611a51e27cbcd59: Log matching property violated. Preceding OpId in replica: term: 2 index: 649. Preceding OpId from leader: term: 2 index: 650. (index mismatch)
I20250110 03:07:03.358960 20668 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 650, Last known committed idx: 649, Time since last communication: 0.000s
I20250110 03:07:03.359081 20868 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 650, Last known committed idx: 649, Time since last communication: 0.000s
I20250110 03:07:03.359751 20668 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Committing config change with OpId 2.650: config changed from index 649 to 650, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.359916 19103 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.650: config changed from index 649 to 650, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.359917 20500 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Committing config change with OpId 2.650: config changed from index 649 to 650, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.361075 17307 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de reported cstate change: config changed from index 649 to 650, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0b345888171f4abcb611a51e27cbcd59" committed_config { opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.419391 20785 raft_consensus.cc:1059] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e: attempting to promote NON_VOTER 674934428b684f929c7b3e408795d6de to VOTER
I20250110 03:07:03.419567 20785 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 2.516, Last appended by leader: 231, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.420034 19103 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 516. Preceding OpId from leader: term: 2 index: 517. (index mismatch)
I20250110 03:07:03.420068 20500 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 LEARNER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 516. Preceding OpId from leader: term: 2 index: 517. (index mismatch)
I20250110 03:07:03.420241 20784 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20250110 03:07:03.420356 20660 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s
I20250110 03:07:03.421072 20785 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.517: config changed from index 516 to 517, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.421136 19103 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Committing config change with OpId 2.517: config changed from index 516 to 517, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.421941 20500 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Committing config change with OpId 2.517: config changed from index 516 to 517, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.422158 17311 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 516 to 517, 674934428b684f929c7b3e408795d6de (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:03.423167 17588 consensus_queue.cc:237] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 517, Committed index: 517, Last appended: 2.517, Last appended by leader: 231, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.423477 20500 raft_consensus.cc:1270] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Refusing update from remote peer 74fcf6c6434b49d8a777aaeda45d971e: Log matching property violated. Preceding OpId in replica: term: 2 index: 517. Preceding OpId from leader: term: 2 index: 518. (index mismatch)
I20250110 03:07:03.423681 20660 consensus_queue.cc:1035] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 518, Last known committed idx: 517, Time since last communication: 0.000s
I20250110 03:07:03.424160 20784 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e [term 2 LEADER]: Committing config change with OpId 2.518: config changed from index 517 to 518, VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) evicted. New config: { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.424311 20500 raft_consensus.cc:2949] T c4d68bc455bd4920ba99280f593678ec P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Committing config change with OpId 2.518: config changed from index 517 to 518, VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) evicted. New config: { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:03.425100 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c4d68bc455bd4920ba99280f593678ec with cas_config_opid_index 517: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:03.425115 17307 catalog_manager.cc:5526] T c4d68bc455bd4920ba99280f593678ec P 74fcf6c6434b49d8a777aaeda45d971e reported cstate change: config changed from index 517 to 518, VOTER d28dc7130e374d8486cc68018c03fd34 (127.14.19.135) evicted. New cstate: current_term: 2 leader_uuid: "74fcf6c6434b49d8a777aaeda45d971e" committed_config { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:03.425879 19084 tablet_service.cc:1514] Processing DeleteTablet for tablet c4d68bc455bd4920ba99280f593678ec with delete_type TABLET_DATA_TOMBSTONED (TS d28dc7130e374d8486cc68018c03fd34 not found in new config with opid_index 518) from {username='slave'} at 127.0.0.1:40260
I20250110 03:07:03.426067 20875 tablet_replica.cc:331] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: stopping tablet replica
I20250110 03:07:03.426148 20875 raft_consensus.cc:2238] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:03.426215 20875 raft_consensus.cc:2267] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:03.427199 20875 ts_tablet_manager.cc:1905] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:03.428344 20875 ts_tablet_manager.cc:1918] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.517
I20250110 03:07:03.428402 20875 log.cc:1198] T c4d68bc455bd4920ba99280f593678ec P d28dc7130e374d8486cc68018c03fd34: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/c4d68bc455bd4920ba99280f593678ec
I20250110 03:07:03.428787 17302 catalog_manager.cc:4872] TS d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779): tablet c4d68bc455bd4920ba99280f593678ec (table rebalance_test_table_2 [id=c21daf0b9b0a4fd5945b0bf7cd1421f5]) successfully deleted
I20250110 03:07:03.461221 17716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37780
I20250110 03:07:03.461314 17716 raft_consensus.cc:604] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Received request to transfer leadership
I20250110 03:07:03.468029 17588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52742
I20250110 03:07:03.468122 17588 raft_consensus.cc:604] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:03.474501 17456 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae"
dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43478
I20250110 03:07:03.474619 17456 raft_consensus.cc:604] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:03.483769 17716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37784
I20250110 03:07:03.483850 17716 raft_consensus.cc:604] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Received request to transfer leadership
I20250110 03:07:03.489643 17588 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0"
dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52778
I20250110 03:07:03.489717 17588 raft_consensus.cc:604] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:03.495530 17456 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae"
dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43492
I20250110 03:07:03.495604 17456 raft_consensus.cc:604] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:03.590669 20785 raft_consensus.cc:988] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: : Instructing follower 92bcfebb99cc47bba372938b64a40f4b to start an election
I20250110 03:07:03.590751 20785 raft_consensus.cc:1076] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Signalling peer 92bcfebb99cc47bba372938b64a40f4b to start an election
I20250110 03:07:03.590924 17456 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0"
dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
from {username='slave'} at 127.14.19.130:56131
I20250110 03:07:03.591006 17456 raft_consensus.cc:491] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:03.591037 17456 raft_consensus.cc:3054] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:03.591645 17456 raft_consensus.cc:513] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.591784 17456 leader_election.cc:290] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 2 election: Requested vote from peers 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185), 674934428b684f929c7b3e408795d6de (127.14.19.136:35499)
I20250110 03:07:03.591914 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: true dest_uuid: "74fcf6c6434b49d8a777aaeda45d971e"
I20250110 03:07:03.592002 17588 raft_consensus.cc:3049] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:03.591967 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fb55ba712c42ee95fcf9f300a54ea0" candidate_uuid: "92bcfebb99cc47bba372938b64a40f4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 441 } } ignore_live_leader: true dest_uuid: "674934428b684f929c7b3e408795d6de"
I20250110 03:07:03.592033 17588 raft_consensus.cc:738] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 74fcf6c6434b49d8a777aaeda45d971e, State: Running, Role: LEADER
I20250110 03:07:03.592051 20500 raft_consensus.cc:3054] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:03.592092 17588 consensus_queue.cc:260] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 441, Committed index: 441, Last appended: 1.441, Last appended by leader: 441, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.592165 17588 raft_consensus.cc:3054] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:03.592839 17588 raft_consensus.cc:2463] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 2.
I20250110 03:07:03.592841 20500 raft_consensus.cc:2463] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92bcfebb99cc47bba372938b64a40f4b in term 2.
I20250110 03:07:03.593016 17392 leader_election.cc:304] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674934428b684f929c7b3e408795d6de, 92bcfebb99cc47bba372938b64a40f4b; no voters:
I20250110 03:07:03.593147 20861 raft_consensus.cc:2798] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:03.593228 20861 raft_consensus.cc:695] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 2 LEADER]: Becoming Leader. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:07:03.593282 20861 consensus_queue.cc:237] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 441, Committed index: 441, Last appended: 1.441, Last appended by leader: 441, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.593789 17307 catalog_manager.cc:5526] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: term changed from 1 to 2, leader changed from 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) to 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129). New cstate: current_term: 2 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 441 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:03.668257 20861 raft_consensus.cc:988] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: : Instructing follower b61b88548de84191b17de06cca3c435a to start an election
I20250110 03:07:03.668334 20861 raft_consensus.cc:1076] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Signalling peer b61b88548de84191b17de06cca3c435a to start an election
I20250110 03:07:03.668545 18705 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae"
dest_uuid: "b61b88548de84191b17de06cca3c435a"
from {username='slave'} at 127.14.19.129:51481
I20250110 03:07:03.668654 18705 raft_consensus.cc:491] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:03.668709 18705 raft_consensus.cc:3054] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:03.669289 18705 raft_consensus.cc:513] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.669521 18705 leader_election.cc:290] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [CANDIDATE]: Term 2 election: Requested vote from peers 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521), 674934428b684f929c7b3e408795d6de (127.14.19.136:35499)
I20250110 03:07:03.669708 17456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae" candidate_uuid: "b61b88548de84191b17de06cca3c435a" candidate_term: 2 candidate_status { last_received { term: 1 index: 518 } } ignore_live_leader: true dest_uuid: "92bcfebb99cc47bba372938b64a40f4b"
I20250110 03:07:03.669826 17456 raft_consensus.cc:3049] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:03.669854 17456 raft_consensus.cc:738] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92bcfebb99cc47bba372938b64a40f4b, State: Running, Role: LEADER
I20250110 03:07:03.669917 17456 consensus_queue.cc:260] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 1.518, Last appended by leader: 518, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.670001 17456 raft_consensus.cc:3054] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:03.670778 17456 raft_consensus.cc:2463] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61b88548de84191b17de06cca3c435a in term 2.
I20250110 03:07:03.671005 18639 leader_election.cc:304] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92bcfebb99cc47bba372938b64a40f4b, b61b88548de84191b17de06cca3c435a; no voters:
I20250110 03:07:03.671105 20866 raft_consensus.cc:2798] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:03.671200 20866 raft_consensus.cc:695] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 2 LEADER]: Becoming Leader. State: Replica: b61b88548de84191b17de06cca3c435a, State: Running, Role: LEADER
I20250110 03:07:03.671269 20866 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 518, Committed index: 518, Last appended: 1.518, Last appended by leader: 518, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.671893 17307 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a reported cstate change: term changed from 1 to 2, leader changed from 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) to b61b88548de84191b17de06cca3c435a (127.14.19.132). New cstate: current_term: 2 leader_uuid: "b61b88548de84191b17de06cca3c435a" committed_config { opid_index: 518 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:03.672967 20500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "593fb2f6a0a7417ca54b800249318eae" candidate_uuid: "b61b88548de84191b17de06cca3c435a" candidate_term: 2 candidate_status { last_received { term: 1 index: 518 } } ignore_live_leader: true dest_uuid: "674934428b684f929c7b3e408795d6de"
I20250110 03:07:03.673051 20500 raft_consensus.cc:3054] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:03.673761 20500 raft_consensus.cc:2463] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b61b88548de84191b17de06cca3c435a in term 2.
I20250110 03:07:03.702116 17716 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37792
I20250110 03:07:03.702190 17716 raft_consensus.cc:604] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Received request to transfer leadership
I20250110 03:07:03.780717 20808 raft_consensus.cc:988] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: : Instructing follower 674934428b684f929c7b3e408795d6de to start an election
I20250110 03:07:03.780783 20808 raft_consensus.cc:1076] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Signalling peer 674934428b684f929c7b3e408795d6de to start an election
I20250110 03:07:03.780951 20500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e365416b475a4288b17d235deddf139d"
dest_uuid: "674934428b684f929c7b3e408795d6de"
from {username='slave'} at 127.14.19.131:51047
I20250110 03:07:03.781040 20500 raft_consensus.cc:491] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:03.781060 20500 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:03.781630 20500 raft_consensus.cc:513] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.781873 20500 leader_election.cc:290] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [CANDIDATE]: Term 3 election: Requested vote from peers 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605), d28dc7130e374d8486cc68018c03fd34 (127.14.19.135:32779)
I20250110 03:07:03.782042 17716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365416b475a4288b17d235deddf139d" candidate_uuid: "674934428b684f929c7b3e408795d6de" candidate_term: 3 candidate_status { last_received { term: 2 index: 650 } } ignore_live_leader: true dest_uuid: "0b345888171f4abcb611a51e27cbcd59"
I20250110 03:07:03.782140 17716 raft_consensus.cc:3049] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Stepping down as leader of term 2
I20250110 03:07:03.782166 17716 raft_consensus.cc:738] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0b345888171f4abcb611a51e27cbcd59, State: Running, Role: LEADER
I20250110 03:07:03.782222 17716 consensus_queue.cc:260] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 650, Committed index: 650, Last appended: 2.650, Last appended by leader: 650, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.782301 17716 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:03.783023 17716 raft_consensus.cc:2463] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674934428b684f929c7b3e408795d6de in term 3.
I20250110 03:07:03.783202 20434 leader_election.cc:304] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b345888171f4abcb611a51e27cbcd59, 674934428b684f929c7b3e408795d6de; no voters:
I20250110 03:07:03.783318 20734 raft_consensus.cc:2798] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 3 FOLLOWER]: Leader election won for term 3
I20250110 03:07:03.783437 20734 raft_consensus.cc:695] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 3 LEADER]: Becoming Leader. State: Replica: 674934428b684f929c7b3e408795d6de, State: Running, Role: LEADER
I20250110 03:07:03.783535 20734 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 650, Committed index: 650, Last appended: 2.650, Last appended by leader: 650, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:03.784134 17307 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de reported cstate change: term changed from 2 to 3, leader changed from 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) to 674934428b684f929c7b3e408795d6de (127.14.19.136). New cstate: current_term: 3 leader_uuid: "674934428b684f929c7b3e408795d6de" committed_config { opid_index: 650 OBSOLETE_local: false peers { permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:03.784987 19103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e365416b475a4288b17d235deddf139d" candidate_uuid: "674934428b684f929c7b3e408795d6de" candidate_term: 3 candidate_status { last_received { term: 2 index: 650 } } ignore_live_leader: true dest_uuid: "d28dc7130e374d8486cc68018c03fd34"
I20250110 03:07:03.785063 19103 raft_consensus.cc:3054] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:03.785589 19103 raft_consensus.cc:2463] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674934428b684f929c7b3e408795d6de in term 3.
I20250110 03:07:04.018844 17588 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 441. Preceding OpId from leader: term: 2 index: 442. (index mismatch)
I20250110 03:07:04.019086 20861 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fcf6c6434b49d8a777aaeda45d971e" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35185 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 442, Last known committed idx: 441, Time since last communication: 0.000s
I20250110 03:07:04.020279 20500 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 441. Preceding OpId from leader: term: 2 index: 442. (index mismatch)
I20250110 03:07:04.020605 20985 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 442, Last known committed idx: 441, Time since last communication: 0.000s
I20250110 03:07:04.086041 17456 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Refusing update from remote peer b61b88548de84191b17de06cca3c435a: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 2 index: 519. (index mismatch)
I20250110 03:07:04.086284 20866 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [LEADER]: Connected to new peer: Peer: permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20250110 03:07:04.087600 20500 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Refusing update from remote peer b61b88548de84191b17de06cca3c435a: Log matching property violated. Preceding OpId in replica: term: 1 index: 518. Preceding OpId from leader: term: 2 index: 519. (index mismatch)
I20250110 03:07:04.087860 20943 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 519, Last known committed idx: 518, Time since last communication: 0.000s
I20250110 03:07:04.186760 19103 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 3 FOLLOWER]: Refusing update from remote peer 674934428b684f929c7b3e408795d6de: Log matching property violated. Preceding OpId in replica: term: 2 index: 650. Preceding OpId from leader: term: 3 index: 651. (index mismatch)
I20250110 03:07:04.187028 20734 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 651, Last known committed idx: 650, Time since last communication: 0.000s
I20250110 03:07:04.188081 17716 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 3 FOLLOWER]: Refusing update from remote peer 674934428b684f929c7b3e408795d6de: Log matching property violated. Preceding OpId in replica: term: 2 index: 650. Preceding OpId from leader: term: 3 index: 651. (index mismatch)
I20250110 03:07:04.188445 21003 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b345888171f4abcb611a51e27cbcd59" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 46605 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 651, Last known committed idx: 650, Time since last communication: 0.000s
I20250110 03:07:04.191130 20500 consensus_queue.cc:237] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 651, Committed index: 651, Last appended: 3.651, Last appended by leader: 650, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 652 OBSOLETE_local: false peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:04.191517 19103 raft_consensus.cc:1270] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 3 FOLLOWER]: Refusing update from remote peer 674934428b684f929c7b3e408795d6de: Log matching property violated. Preceding OpId in replica: term: 3 index: 651. Preceding OpId from leader: term: 3 index: 652. (index mismatch)
I20250110 03:07:04.191696 21003 consensus_queue.cc:1035] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 652, Last known committed idx: 651, Time since last communication: 0.000s
I20250110 03:07:04.192165 20734 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P 674934428b684f929c7b3e408795d6de [term 3 LEADER]: Committing config change with OpId 3.652: config changed from index 650 to 652, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 652 OBSOLETE_local: false peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.192288 19103 raft_consensus.cc:2949] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 [term 3 FOLLOWER]: Committing config change with OpId 3.652: config changed from index 650 to 652, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New config: { opid_index: 652 OBSOLETE_local: false peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.193068 17301 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e365416b475a4288b17d235deddf139d with cas_config_opid_index 650: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:04.193197 17307 catalog_manager.cc:5526] T e365416b475a4288b17d235deddf139d P d28dc7130e374d8486cc68018c03fd34 reported cstate change: config changed from index 650 to 652, VOTER 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131) evicted. New cstate: current_term: 3 leader_uuid: "674934428b684f929c7b3e408795d6de" committed_config { opid_index: 652 OBSOLETE_local: false peers { permanent_uuid: "d28dc7130e374d8486cc68018c03fd34" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 32779 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.193958 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet e365416b475a4288b17d235deddf139d with delete_type TABLET_DATA_TOMBSTONED (TS 0b345888171f4abcb611a51e27cbcd59 not found in new config with opid_index 652) from {username='slave'} at 127.0.0.1:40906
I20250110 03:07:04.194128 21007 tablet_replica.cc:331] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: stopping tablet replica
I20250110 03:07:04.194187 21007 raft_consensus.cc:2238] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:04.194228 21007 raft_consensus.cc:2267] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:04.195123 21007 ts_tablet_manager.cc:1905] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:04.196557 21007 ts_tablet_manager.cc:1918] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.651
I20250110 03:07:04.196614 21007 log.cc:1198] T e365416b475a4288b17d235deddf139d P 0b345888171f4abcb611a51e27cbcd59: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/e365416b475a4288b17d235deddf139d
I20250110 03:07:04.197008 17300 catalog_manager.cc:4872] TS 0b345888171f4abcb611a51e27cbcd59 (127.14.19.131:46605): tablet e365416b475a4288b17d235deddf139d (table rebalance_test_table_1 [id=96472a36863f4aac8800fef1a5b0164a]) successfully deleted
I20250110 03:07:04.453095 17456 consensus_queue.cc:237] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 442, Committed index: 442, Last appended: 2.442, Last appended by leader: 441, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:04.453490 20500 raft_consensus.cc:1270] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Refusing update from remote peer 92bcfebb99cc47bba372938b64a40f4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 442. Preceding OpId from leader: term: 2 index: 443. (index mismatch)
I20250110 03:07:04.453714 20861 consensus_queue.cc:1035] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 443, Last known committed idx: 442, Time since last communication: 0.000s
I20250110 03:07:04.454368 20985 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b [term 2 LEADER]: Committing config change with OpId 2.443: config changed from index 441 to 443, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.454510 20500 raft_consensus.cc:2949] T 10fb55ba712c42ee95fcf9f300a54ea0 P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Committing config change with OpId 2.443: config changed from index 441 to 443, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New config: { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.455430 17299 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 10fb55ba712c42ee95fcf9f300a54ea0 with cas_config_opid_index 441: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:04.455678 17307 catalog_manager.cc:5526] T 10fb55ba712c42ee95fcf9f300a54ea0 P 92bcfebb99cc47bba372938b64a40f4b reported cstate change: config changed from index 441 to 443, VOTER 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "92bcfebb99cc47bba372938b64a40f4b" committed_config { opid_index: 443 OBSOLETE_local: false peers { permanent_uuid: "92bcfebb99cc47bba372938b64a40f4b" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:04.456383 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fb55ba712c42ee95fcf9f300a54ea0 with delete_type TABLET_DATA_TOMBSTONED (TS 74fcf6c6434b49d8a777aaeda45d971e not found in new config with opid_index 443) from {username='slave'} at 127.0.0.1:57906
I20250110 03:07:04.456565 21033 tablet_replica.cc:331] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: stopping tablet replica
I20250110 03:07:04.456634 21033 raft_consensus.cc:2238] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:04.456679 21033 raft_consensus.cc:2267] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:04.457501 21033 ts_tablet_manager.cc:1905] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:04.458796 21033 ts_tablet_manager.cc:1918] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.442
I20250110 03:07:04.458849 21033 log.cc:1198] T 10fb55ba712c42ee95fcf9f300a54ea0 P 74fcf6c6434b49d8a777aaeda45d971e: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/10fb55ba712c42ee95fcf9f300a54ea0
I20250110 03:07:04.459204 17300 catalog_manager.cc:4872] TS 74fcf6c6434b49d8a777aaeda45d971e (127.14.19.130:35185): tablet 10fb55ba712c42ee95fcf9f300a54ea0 (table rebalance_test_table_4 [id=2956ba1352714102a14ee5ea173d59fa]) successfully deleted
I20250110 03:07:04.553140 18705 consensus_queue.cc:237] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 519, Committed index: 519, Last appended: 2.519, Last appended by leader: 518, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } }
I20250110 03:07:04.553611 20500 raft_consensus.cc:1270] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Refusing update from remote peer b61b88548de84191b17de06cca3c435a: Log matching property violated. Preceding OpId in replica: term: 2 index: 519. Preceding OpId from leader: term: 2 index: 520. (index mismatch)
I20250110 03:07:04.553791 20866 consensus_queue.cc:1035] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [LEADER]: Connected to new peer: Peer: permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 520, Last known committed idx: 519, Time since last communication: 0.000s
I20250110 03:07:04.554394 20943 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a [term 2 LEADER]: Committing config change with OpId 2.520: config changed from index 518 to 520, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.554529 20500 raft_consensus.cc:2949] T 593fb2f6a0a7417ca54b800249318eae P 674934428b684f929c7b3e408795d6de [term 2 FOLLOWER]: Committing config change with OpId 2.520: config changed from index 518 to 520, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New config: { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } } }
I20250110 03:07:04.555387 17300 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 593fb2f6a0a7417ca54b800249318eae with cas_config_opid_index 518: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:04.555651 17307 catalog_manager.cc:5526] T 593fb2f6a0a7417ca54b800249318eae P b61b88548de84191b17de06cca3c435a reported cstate change: config changed from index 518 to 520, VOTER 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "b61b88548de84191b17de06cca3c435a" committed_config { opid_index: 520 OBSOLETE_local: false peers { permanent_uuid: "b61b88548de84191b17de06cca3c435a" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38659 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "674934428b684f929c7b3e408795d6de" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 35499 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:04.556535 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet 593fb2f6a0a7417ca54b800249318eae with delete_type TABLET_DATA_TOMBSTONED (TS 92bcfebb99cc47bba372938b64a40f4b not found in new config with opid_index 520) from {username='slave'} at 127.0.0.1:60766
I20250110 03:07:04.556744 21052 tablet_replica.cc:331] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: stopping tablet replica
I20250110 03:07:04.556818 21052 raft_consensus.cc:2238] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:04.556865 21052 raft_consensus.cc:2267] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:04.558053 21052 ts_tablet_manager.cc:1905] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:04.559443 21052 ts_tablet_manager.cc:1918] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.519
I20250110 03:07:04.559506 21052 log.cc:1198] T 593fb2f6a0a7417ca54b800249318eae P 92bcfebb99cc47bba372938b64a40f4b: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/593fb2f6a0a7417ca54b800249318eae
I20250110 03:07:04.559950 17299 catalog_manager.cc:4872] TS 92bcfebb99cc47bba372938b64a40f4b (127.14.19.129:42521): tablet 593fb2f6a0a7417ca54b800249318eae (table rebalance_test_table_3 [id=547950e4f5174f7c8de141cb25c6be98]) successfully deleted
I20250110 03:07:04.580672 17697 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:04.581215 20480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.583622 17568 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:04.588860 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:04.590755 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.593147 18686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.593818 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:04.597926 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.628486 17697 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:04.631292 20480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.631989 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:04.633455 17568 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:04.638468 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.640942 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.645375 18686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.650207 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:04.699024 18949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:04.699173 20480 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.701365 18686 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.708093 17435 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:04.708348 18817 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:04.714859 17697 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:04.716780 17568 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:04.717096 19084 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7c800037a95e4de6bd43d671ea55076b | 127.14.19.190:35275 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:42975 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0b345888171f4abcb611a51e27cbcd59 | 127.14.19.131:46605 | HEALTHY | <none> | 2 | 0
4ce3080cc7de4c729423b95fb7158a07 | 127.14.19.134:36693 | HEALTHY | <none> | 2 | 0
674934428b684f929c7b3e408795d6de | 127.14.19.136:35499 | HEALTHY | <none> | 1 | 0
74fcf6c6434b49d8a777aaeda45d971e | 127.14.19.130:35185 | HEALTHY | <none> | 4 | 0
92bcfebb99cc47bba372938b64a40f4b | 127.14.19.129:42521 | HEALTHY | <none> | 4 | 0
b61b88548de84191b17de06cca3c435a | 127.14.19.132:38659 | HEALTHY | <none> | 1 | 0
d1e6d3913b04475f8962dd6496fbea0a | 127.14.19.133:39385 | HEALTHY | <none> | 1 | 0
d28dc7130e374d8486cc68018c03fd34 | 127.14.19.135:32779 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.14.19.129 | experimental | 127.14.19.129:42521
local_ip_for_outbound_sockets | 127.14.19.130 | experimental | 127.14.19.130:35185
local_ip_for_outbound_sockets | 127.14.19.131 | experimental | 127.14.19.131:46605
local_ip_for_outbound_sockets | 127.14.19.132 | experimental | 127.14.19.132:38659
local_ip_for_outbound_sockets | 127.14.19.133 | experimental | 127.14.19.133:39385
local_ip_for_outbound_sockets | 127.14.19.134 | experimental | 127.14.19.134:36693
local_ip_for_outbound_sockets | 127.14.19.135 | experimental | 127.14.19.135:32779
local_ip_for_outbound_sockets | 127.14.19.136 | experimental | 127.14.19.136:35499
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.19.129:42521
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.19.130:35185
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.19.131:46605
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.19.132:38659
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.19.133:39385
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.19.134:36693
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.19.135:32779
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalanceParamTest.Rebalance_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.14.19.136:35499
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:42975 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 03:07:04.727607 14414 log_verifier.cc:126] Checking tablet 10fb55ba712c42ee95fcf9f300a54ea0
I20250110 03:07:04.753278 14414 log_verifier.cc:177] Verified matching terms for 443 ops in tablet 10fb55ba712c42ee95fcf9f300a54ea0
I20250110 03:07:04.753337 14414 log_verifier.cc:126] Checking tablet 567885520d33460fae980ae73fabcd6a
I20250110 03:07:04.774513 14414 log_verifier.cc:177] Verified matching terms for 437 ops in tablet 567885520d33460fae980ae73fabcd6a
I20250110 03:07:04.774596 14414 log_verifier.cc:126] Checking tablet 593fb2f6a0a7417ca54b800249318eae
I20250110 03:07:04.800969 14414 log_verifier.cc:177] Verified matching terms for 520 ops in tablet 593fb2f6a0a7417ca54b800249318eae
I20250110 03:07:04.801031 14414 log_verifier.cc:126] Checking tablet 701c561086ce457e883794c11cb34ab6
I20250110 03:07:04.830467 14414 log_verifier.cc:177] Verified matching terms for 680 ops in tablet 701c561086ce457e883794c11cb34ab6
I20250110 03:07:04.830538 14414 log_verifier.cc:126] Checking tablet 7906c51301244bd4a592d8e01e6ad602
I20250110 03:07:04.851845 14414 log_verifier.cc:177] Verified matching terms for 421 ops in tablet 7906c51301244bd4a592d8e01e6ad602
I20250110 03:07:04.851904 14414 log_verifier.cc:126] Checking tablet 83d2ceb09bc5442f8c329ca6b3ad6f35
I20250110 03:07:04.872170 14414 log_verifier.cc:177] Verified matching terms for 414 ops in tablet 83d2ceb09bc5442f8c329ca6b3ad6f35
I20250110 03:07:04.872226 14414 log_verifier.cc:126] Checking tablet 85fe921f114943c896a041599c39dbe5
I20250110 03:07:04.894071 14414 log_verifier.cc:177] Verified matching terms for 467 ops in tablet 85fe921f114943c896a041599c39dbe5
I20250110 03:07:04.894130 14414 log_verifier.cc:126] Checking tablet 888f2d12f3254197a6a490e23016bd46
I20250110 03:07:04.899646 14414 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 888f2d12f3254197a6a490e23016bd46
I20250110 03:07:04.899689 14414 log_verifier.cc:126] Checking tablet 8b2221475806475c83aa9867a548f51f
I20250110 03:07:04.922914 14414 log_verifier.cc:177] Verified matching terms for 484 ops in tablet 8b2221475806475c83aa9867a548f51f
I20250110 03:07:04.922971 14414 log_verifier.cc:126] Checking tablet 8dec030e45a84eb89321f275d0013826
I20250110 03:07:04.947347 14414 log_verifier.cc:177] Verified matching terms for 529 ops in tablet 8dec030e45a84eb89321f275d0013826
I20250110 03:07:04.947409 14414 log_verifier.cc:126] Checking tablet c4d68bc455bd4920ba99280f593678ec
I20250110 03:07:04.974051 14414 log_verifier.cc:177] Verified matching terms for 518 ops in tablet c4d68bc455bd4920ba99280f593678ec
I20250110 03:07:04.974146 14414 log_verifier.cc:126] Checking tablet c8b6b5410b4e4405b419e22daf379c6a
I20250110 03:07:04.996318 14414 log_verifier.cc:177] Verified matching terms for 462 ops in tablet c8b6b5410b4e4405b419e22daf379c6a
I20250110 03:07:04.996372 14414 log_verifier.cc:126] Checking tablet e365416b475a4288b17d235deddf139d
I20250110 03:07:05.027603 14414 log_verifier.cc:177] Verified matching terms for 652 ops in tablet e365416b475a4288b17d235deddf139d
I20250110 03:07:05.027673 14414 log_verifier.cc:126] Checking tablet f3ec6ddcb58d48d89f4cea0b3db80035
I20250110 03:07:05.052335 14414 log_verifier.cc:177] Verified matching terms for 518 ops in tablet f3ec6ddcb58d48d89f4cea0b3db80035
I20250110 03:07:05.052440 14414 log_verifier.cc:126] Checking tablet f8b5177de1fa4e1c9668453452f2064f
I20250110 03:07:05.085299 14414 log_verifier.cc:177] Verified matching terms for 657 ops in tablet f8b5177de1fa4e1c9668453452f2064f
I20250110 03:07:05.085368 14414 log_verifier.cc:126] Checking tablet fa58142d4f86446b99561d0a0fb9fc96
I20250110 03:07:05.108685 14414 log_verifier.cc:177] Verified matching terms for 426 ops in tablet fa58142d4f86446b99561d0a0fb9fc96
I20250110 03:07:05.109351 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 17375
I20250110 03:07:05.116168 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 17506
I20250110 03:07:05.121757 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 17637
I20250110 03:07:05.127226 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 18624
I20250110 03:07:05.131824 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 18755
I20250110 03:07:05.136425 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 18886
I20250110 03:07:05.140795 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 19019
I20250110 03:07:05.145246 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 20418
I20250110 03:07:05.149560 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 17284
2025-01-10T03:07:05Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (19329 ms)
[----------] 1 test from RebalanceParamTest (19329 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20250110 03:07:05.200557 14414 test_util.cc:274] Using random seed: 1672591273
I20250110 03:07:05.201025 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:07:05.201054 14414 ts_itest-base.cc:113] --------------
I20250110 03:07:05.201066 14414 ts_itest-base.cc:114] 8 tablet servers
I20250110 03:07:05.201076 14414 ts_itest-base.cc:115] 3 replicas per TS
I20250110 03:07:05.201089 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:07:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:07:05Z Disabled control of system clock
I20250110 03:07:05.208093 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:41945
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:41945
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20250110 03:07:05.279969 21165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:05.280157 21165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:05.280176 21165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:05.281452 21165 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:05.281497 21165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:07:05.281509 21165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:05.281520 21165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:07:05.281530 21165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:07:05.282936 21165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:41945
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:05.283102 21165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:05.283286 21165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:05.287847 21173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.288537 21170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.289762 21165 server_base.cc:1034] running on GCE node
W20250110 03:07:05.290439 21171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.290757 21165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:05.291018 21165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:05.294225 21165 hybrid_clock.cc:648] HybridClock initialized: now 1736478425294209 us; error 29 us; skew 500 ppm
I20250110 03:07:05.295666 21165 webserver.cc:458] Webserver started at http://127.14.19.190:43639/ using document root <none> and password file <none>
I20250110 03:07:05.295886 21165 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:05.295970 21165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:05.296087 21165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:05.297282 21165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f0d1744b334c4bae9f757319e7a75e38"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.297703 21165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f0d1744b334c4bae9f757319e7a75e38"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.299269 21165 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20250110 03:07:05.300041 21179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.300213 21165 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:05.300289 21165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f0d1744b334c4bae9f757319e7a75e38"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.300349 21165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:05.327277 21165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:05.327601 21165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:05.327723 21165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:05.331933 21165 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:41945
I20250110 03:07:05.331959 21231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:41945 every 8 connection(s)
I20250110 03:07:05.332233 21165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:07:05.332813 21232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:05.335191 21232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38: Bootstrap starting.
I20250110 03:07:05.335748 21232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:05.335966 21232 log.cc:826] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:05.336463 21232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38: No bootstrap required, opened a new log
I20250110 03:07:05.337983 21232 raft_consensus.cc:357] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d1744b334c4bae9f757319e7a75e38" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41945 } }
I20250110 03:07:05.338093 21232 raft_consensus.cc:383] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:05.338114 21232 raft_consensus.cc:738] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d1744b334c4bae9f757319e7a75e38, State: Initialized, Role: FOLLOWER
I20250110 03:07:05.338229 21232 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d1744b334c4bae9f757319e7a75e38" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41945 } }
I20250110 03:07:05.338292 21232 raft_consensus.cc:397] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:05.338323 21232 raft_consensus.cc:491] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:05.338371 21232 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:05.338912 21232 raft_consensus.cc:513] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d1744b334c4bae9f757319e7a75e38" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41945 } }
I20250110 03:07:05.339011 21232 leader_election.cc:304] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d1744b334c4bae9f757319e7a75e38; no voters:
I20250110 03:07:05.339145 21232 leader_election.cc:290] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:05.339188 21235 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:05.339311 21235 raft_consensus.cc:695] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [term 1 LEADER]: Becoming Leader. State: Replica: f0d1744b334c4bae9f757319e7a75e38, State: Running, Role: LEADER
I20250110 03:07:05.339334 21232 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:07:05.339392 21235 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d1744b334c4bae9f757319e7a75e38" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41945 } }
I20250110 03:07:05.339692 21237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d1744b334c4bae9f757319e7a75e38. Latest consensus state: current_term: 1 leader_uuid: "f0d1744b334c4bae9f757319e7a75e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d1744b334c4bae9f757319e7a75e38" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41945 } } }
I20250110 03:07:05.339778 21237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:05.339941 21236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d1744b334c4bae9f757319e7a75e38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d1744b334c4bae9f757319e7a75e38" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 41945 } } }
I20250110 03:07:05.340001 21236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:05.340122 21242 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:07:05.340472 21242 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:07:05.341682 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21165
I20250110 03:07:05.341784 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:07:05.342089 21242 catalog_manager.cc:1348] Generated new cluster ID: a7110a9837c447ceacec44ce47230d64
I20250110 03:07:05.342157 21242 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:07:05.359593 21242 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:07:05.360237 21242 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:07:05.366694 21242 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f0d1744b334c4bae9f757319e7a75e38: Generated new TSK 0
I20250110 03:07:05.366847 21242 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:07:05.370579 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:05.446437 21256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:05.446642 21256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:05.446672 21256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:05.448496 21256 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:05.448549 21256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:05.448588 21256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:07:05.450661 21256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.21256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:05.450853 21256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:05.451082 21256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:05.456444 21262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.457420 21264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.458264 21261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.459158 21256 server_base.cc:1034] running on GCE node
I20250110 03:07:05.459299 21256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:05.459468 21256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:05.460592 21256 hybrid_clock.cc:648] HybridClock initialized: now 1736478425460579 us; error 27 us; skew 500 ppm
I20250110 03:07:05.461771 21256 webserver.cc:458] Webserver started at http://127.14.19.129:41477/ using document root <none> and password file <none>
I20250110 03:07:05.461944 21256 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:05.461989 21256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:05.462078 21256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:05.462903 21256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.463182 21256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.464313 21256 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:05.464927 21270 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.465086 21256 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:05.465170 21256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.465230 21256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:05.489706 21256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:05.489926 21256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:05.490025 21256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:05.490186 21256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:05.490478 21256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:05.490514 21256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.490540 21256 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:05.490593 21256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.497134 21256 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:45257
I20250110 03:07:05.497166 21383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:45257 every 8 connection(s)
I20250110 03:07:05.497440 21256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:07:05.502471 21384 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:05.502575 21384 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:05.502744 21384 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:05.503172 21196 ts_manager.cc:194] Registered new tserver with Master: a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:05.503767 21196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:36921
I20250110 03:07:05.507124 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21256
I20250110 03:07:05.507206 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:07:05.508121 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:05.579300 21387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:05.579458 21387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:05.579478 21387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:05.580669 21387 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:05.580719 21387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:05.580739 21387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:07:05.581990 21387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.21387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:05.582156 21387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:05.582346 21387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:05.586616 21392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.587463 21395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.588274 21393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.589210 21387 server_base.cc:1034] running on GCE node
I20250110 03:07:05.589324 21387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:05.589488 21387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:05.590600 21387 hybrid_clock.cc:648] HybridClock initialized: now 1736478425590584 us; error 33 us; skew 500 ppm
I20250110 03:07:05.591598 21387 webserver.cc:458] Webserver started at http://127.14.19.130:40643/ using document root <none> and password file <none>
I20250110 03:07:05.591742 21387 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:05.591779 21387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:05.591841 21387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:05.592643 21387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ab3ed3d8c2394f78943461e22853c7ac"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.592906 21387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ab3ed3d8c2394f78943461e22853c7ac"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.593935 21387 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:05.594451 21401 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.594614 21387 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.594669 21387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ab3ed3d8c2394f78943461e22853c7ac"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.594709 21387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:05.620321 21387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:05.620503 21387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:05.620589 21387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:05.620714 21387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:05.620990 21387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:05.621021 21387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.621042 21387 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:05.621066 21387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.626084 21387 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:35469
I20250110 03:07:05.626142 21514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:35469 every 8 connection(s)
I20250110 03:07:05.626377 21387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:07:05.630489 21515 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:05.630594 21515 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:05.630795 21515 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:05.631099 21196 ts_manager.cc:194] Registered new tserver with Master: ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:05.631433 21196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:32855
I20250110 03:07:05.631935 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21387
I20250110 03:07:05.632004 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:07:05.632807 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:05.705937 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:05.706073 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:05.706090 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:05.707276 21518 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:05.707312 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:05.707333 21518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:07:05.708665 21518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:05.708819 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:05.709000 21518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:05.713440 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.714375 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.715229 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.716080 21518 server_base.cc:1034] running on GCE node
I20250110 03:07:05.716224 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:05.716408 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:05.717521 21518 hybrid_clock.cc:648] HybridClock initialized: now 1736478425717507 us; error 29 us; skew 500 ppm
I20250110 03:07:05.718636 21518 webserver.cc:458] Webserver started at http://127.14.19.131:39927/ using document root <none> and password file <none>
I20250110 03:07:05.718822 21518 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:05.718869 21518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:05.718953 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:05.719743 21518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.720027 21518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.721076 21518 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:05.721616 21532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.721752 21518 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:05.721822 21518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.721874 21518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:05.737697 21518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:05.737920 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:05.738021 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:05.738173 21518 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:05.738430 21518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:05.738461 21518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.738489 21518 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:05.738514 21518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.743801 21518 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:38011
I20250110 03:07:05.743847 21645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:38011 every 8 connection(s)
I20250110 03:07:05.744091 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:07:05.746498 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21518
I20250110 03:07:05.746608 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:07:05.747638 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:05.748729 21646 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:05.748821 21646 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:05.749060 21646 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:05.749408 21196 ts_manager.cc:194] Registered new tserver with Master: c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:05.749902 21196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:46473
W20250110 03:07:05.819581 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:05.819727 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:05.819746 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:05.820911 21649 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:05.820946 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:05.820978 21649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:05.822187 21649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.21649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:05.822360 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:05.822628 21649 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:05.827227 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.828182 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.828943 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.829838 21649 server_base.cc:1034] running on GCE node
I20250110 03:07:05.829973 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:05.830122 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:05.831226 21649 hybrid_clock.cc:648] HybridClock initialized: now 1736478425831216 us; error 27 us; skew 500 ppm
I20250110 03:07:05.832270 21649 webserver.cc:458] Webserver started at http://127.14.19.132:39125/ using document root <none> and password file <none>
I20250110 03:07:05.832453 21649 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:05.832500 21649 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:05.832599 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:05.833387 21649 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.833673 21649 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.834739 21649 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:05.835335 21663 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.835474 21649 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.835537 21649 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.835590 21649 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:05.853058 21649 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:05.853242 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:05.853336 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:05.853494 21649 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:05.853756 21649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:05.853793 21649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.853822 21649 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:05.853849 21649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.858920 21649 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:45355
I20250110 03:07:05.858978 21776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:45355 every 8 connection(s)
I20250110 03:07:05.859215 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:05.861984 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21649
I20250110 03:07:05.862059 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:07:05.862983 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:0
--local_ip_for_outbound_sockets=127.14.19.133
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:05.864797 21777 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:05.864872 21777 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:05.865037 21777 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:05.865324 21196 ts_manager.cc:194] Registered new tserver with Master: 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:05.865690 21196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:60053
W20250110 03:07:05.936139 21780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:05.936288 21780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:05.936316 21780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:05.937484 21780 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:05.937522 21780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:05.937553 21780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:07:05.938948 21780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.21780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:05.939175 21780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:05.939350 21780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:05.943665 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.944676 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:05.945363 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:05.946292 21780 server_base.cc:1034] running on GCE node
I20250110 03:07:05.946476 21780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:05.946708 21780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:05.947826 21780 hybrid_clock.cc:648] HybridClock initialized: now 1736478425947815 us; error 25 us; skew 500 ppm
I20250110 03:07:05.948885 21780 webserver.cc:458] Webserver started at http://127.14.19.133:41347/ using document root <none> and password file <none>
I20250110 03:07:05.949079 21780 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:05.949121 21780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:05.949208 21780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:05.949998 21780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e4d83809ce4746d5ad65e8462857fd9c"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.950268 21780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e4d83809ce4746d5ad65e8462857fd9c"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.951378 21780 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:05.951896 21794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.952023 21780 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:05.952082 21780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e4d83809ce4746d5ad65e8462857fd9c"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:05.952129 21780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:05.968983 21780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:05.969190 21780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:05.969280 21780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:05.969430 21780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:05.969683 21780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:05.969714 21780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.969743 21780 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:05.969770 21780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:05.974965 21780 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:42091
I20250110 03:07:05.975011 21907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:42091 every 8 connection(s)
I20250110 03:07:05.975243 21780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:07:05.976838 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21780
I20250110 03:07:05.976928 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 03:07:05.977908 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:0
--local_ip_for_outbound_sockets=127.14.19.134
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:05.979718 21908 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:05.979808 21908 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:05.979997 21908 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:05.980297 21196 ts_manager.cc:194] Registered new tserver with Master: e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:05.980651 21196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:55601
W20250110 03:07:06.051036 21911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:06.051184 21911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:06.051215 21911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:06.052420 21911 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:06.052462 21911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:06.052498 21911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:07:06.053961 21911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.21911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:06.054150 21911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:06.054337 21911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:06.059024 21916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:06.060022 21919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:06.060869 21917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:06.061760 21911 server_base.cc:1034] running on GCE node
I20250110 03:07:06.061904 21911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:06.062074 21911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:06.063189 21911 hybrid_clock.cc:648] HybridClock initialized: now 1736478426063174 us; error 30 us; skew 500 ppm
I20250110 03:07:06.064200 21911 webserver.cc:458] Webserver started at http://127.14.19.134:41101/ using document root <none> and password file <none>
I20250110 03:07:06.064381 21911 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:06.064426 21911 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:06.064528 21911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:06.065320 21911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "531cf07bcce6435c9e18185c60b2118d"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.065634 21911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "531cf07bcce6435c9e18185c60b2118d"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.066706 21911 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:06.067271 21925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.067426 21911 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.067490 21911 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "531cf07bcce6435c9e18185c60b2118d"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.067543 21911 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:06.083683 21911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:06.083873 21911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:06.083966 21911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:06.084110 21911 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:06.084353 21911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:06.084383 21911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.084411 21911 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:06.084439 21911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.089452 21911 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:45351
I20250110 03:07:06.089502 22038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:45351 every 8 connection(s)
I20250110 03:07:06.089708 21911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:07:06.092303 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 21911
I20250110 03:07:06.092424 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250110 03:07:06.093533 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.135:0
--local_ip_for_outbound_sockets=127.14.19.135
--webserver_interface=127.14.19.135
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:06.093919 22039 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:06.094012 22039 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:06.094225 22039 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:06.094589 21196 ts_manager.cc:194] Registered new tserver with Master: 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351)
I20250110 03:07:06.095068 21196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:37243
W20250110 03:07:06.167413 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:06.167559 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:06.167578 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:06.168752 22042 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:06.168787 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:06.168808 22042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.135
I20250110 03:07:06.170085 22042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.19.135
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.22042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.135
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:06.170305 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:06.170485 22042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:06.174850 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:06.175720 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:06.176584 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:06.177345 22042 server_base.cc:1034] running on GCE node
I20250110 03:07:06.177486 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:06.177661 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:06.178774 22042 hybrid_clock.cc:648] HybridClock initialized: now 1736478426178766 us; error 39 us; skew 500 ppm
I20250110 03:07:06.179788 22042 webserver.cc:458] Webserver started at http://127.14.19.135:43769/ using document root <none> and password file <none>
I20250110 03:07:06.179965 22042 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:06.180002 22042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:06.180075 22042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:06.180845 22042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b2e7bce763e24b63bc760b0b82b53454"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.181097 22042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b2e7bce763e24b63bc760b0b82b53454"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.182188 22042 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:06.182729 22056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.182888 22042 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.182950 22042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b2e7bce763e24b63bc760b0b82b53454"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.182993 22042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:06.198457 22042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:06.198652 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:06.198740 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:06.198872 22042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:06.199101 22042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:06.199127 22042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.199151 22042 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:06.199167 22042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.204051 22042 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.135:39887
I20250110 03:07:06.204090 22169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.135:39887 every 8 connection(s)
I20250110 03:07:06.204310 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250110 03:07:06.207949 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 22042
I20250110 03:07:06.208043 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20250110 03:07:06.208591 22170 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:06.208662 22170 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:06.208820 22170 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:06.209143 21195 ts_manager.cc:194] Registered new tserver with Master: b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887)
I20250110 03:07:06.209326 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.136:0
--local_ip_for_outbound_sockets=127.14.19.136
--webserver_interface=127.14.19.136
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:06.209564 21195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.135:48821
W20250110 03:07:06.281388 22173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:06.281543 22173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:06.281572 22173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:06.282781 22173 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:06.282816 22173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:06.282846 22173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.136
I20250110 03:07:06.284226 22173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.19.136
--webserver_port=0
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.22173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.136
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:06.284417 22173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:06.284705 22173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:06.289378 22178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:06.290153 22181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:06.291021 22179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:06.291855 22173 server_base.cc:1034] running on GCE node
I20250110 03:07:06.291985 22173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:06.292135 22173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:06.293248 22173 hybrid_clock.cc:648] HybridClock initialized: now 1736478426293234 us; error 31 us; skew 500 ppm
I20250110 03:07:06.294293 22173 webserver.cc:458] Webserver started at http://127.14.19.136:41375/ using document root <none> and password file <none>
I20250110 03:07:06.294458 22173 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:06.294498 22173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:06.294617 22173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:06.295394 22173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.295657 22173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.296705 22173 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:06.297264 22187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.297390 22173 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:06.297449 22173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:06.297500 22173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:06.311044 22173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:06.311208 22173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:06.311295 22173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:06.311434 22173 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:06.311681 22173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:06.311710 22173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.311739 22173 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:06.311769 22173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:06.316830 22173 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.136:42591
I20250110 03:07:06.316886 22300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.136:42591 every 8 connection(s)
I20250110 03:07:06.317113 22173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20250110 03:07:06.321408 22301 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:06.321475 22301 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:06.321614 22301 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:06.321899 21195 ts_manager.cc:194] Registered new tserver with Master: d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:06.322201 21195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.136:60477
I20250110 03:07:06.323424 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 22173
I20250110 03:07:06.323515 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20250110 03:07:06.324365 14414 external_mini_cluster.cc:934] 8 TS(s) registered with all masters
I20250110 03:07:06.329659 14414 external_mini_cluster.cc:934] 8 TS(s) registered with all masters
I20250110 03:07:06.329705 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21780
I20250110 03:07:06.334424 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21911
I20250110 03:07:06.338320 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 22042
I20250110 03:07:06.343312 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 22173
I20250110 03:07:06.504467 21384 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:06.632112 21515 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:06.750532 21646 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:06.866500 21777 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:10.097833 21195 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:10.106067 21711 tablet_service.cc:1467] Processing CreateTablet for tablet 250f547c54a84a35b2850d50b897810e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.106067 21579 tablet_service.cc:1467] Processing CreateTablet for tablet c5d0922741634918830ee2c163cbce24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.106372 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.106372 21711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.106242 21316 tablet_service.cc:1467] Processing CreateTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.106242 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 250f547c54a84a35b2850d50b897810e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.106516 21317 tablet_service.cc:1467] Processing CreateTablet for tablet c5d0922741634918830ee2c163cbce24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.106918 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.106981 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 250f547c54a84a35b2850d50b897810e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.107089 21580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.107342 21710 tablet_service.cc:1467] Processing CreateTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.107478 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.108400 21317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.109853 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.110096 22323 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.110702 22323 tablet_bootstrap.cc:654] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.110791 21448 tablet_service.cc:1467] Processing CreateTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.111029 22323 log.cc:826] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:10.111043 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.111040 21449 tablet_service.cc:1467] Processing CreateTablet for tablet c5d0922741634918830ee2c163cbce24 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.111145 22321 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.111387 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.111678 22323 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.111717 22321 tablet_bootstrap.cc:654] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.111735 22323 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:10.111986 22321 log.cc:826] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:10.112282 22322 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:10.112850 22322 tablet_bootstrap.cc:654] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.113109 22322 log.cc:826] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:10.113883 22321 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.113794 22323 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.113931 22323 raft_consensus.cc:383] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.113968 22323 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.114048 22323 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.114202 22322 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:10.114109 22321 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:10.114270 22322 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:10.114291 22323 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:07:10.114394 22323 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.114405 22327 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.114857 22323 tablet_bootstrap.cc:654] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.114979 22327 tablet_bootstrap.cc:654] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.115343 22327 log.cc:826] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:10.115767 22323 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.115811 22323 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.115942 22323 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.116003 22323 raft_consensus.cc:383] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.116021 22327 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.116027 22323 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.116070 22327 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:10.116073 22323 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.116143 22323 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.116184 22323 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.116577 22323 tablet_bootstrap.cc:654] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.116741 22322 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.116900 22322 raft_consensus.cc:383] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.116938 22322 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.117029 22322 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.117184 22323 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.117228 22323 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.117240 22322 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:07:10.117326 22322 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:10.117352 22323 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.117410 22323 raft_consensus.cc:383] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.117434 22323 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.117480 22323 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.117693 22323 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.117728 22322 tablet_bootstrap.cc:654] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.118459 22322 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:10.118405 22327 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.118501 22322 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:10.118523 22327 raft_consensus.cc:383] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.118554 22327 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.118659 22322 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.118661 22327 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.118718 22322 raft_consensus.cc:383] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.118741 22322 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.118788 22322 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.118894 22327 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250110 03:07:10.118923 22322 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:10.119165 22327 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.119530 22321 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.119644 22321 raft_consensus.cc:383] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.119639 22327 tablet_bootstrap.cc:654] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.119678 22321 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.119758 22321 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.120003 22321 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250110 03:07:10.120086 22321 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.120223 22327 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.120271 22327 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.120411 22327 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.120471 22327 raft_consensus.cc:383] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.120496 22327 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.120525 22321 tablet_bootstrap.cc:654] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.120537 22327 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.120649 22327 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.121109 22321 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.121148 22321 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:10.121258 22321 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.121296 22321 raft_consensus.cc:383] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.121317 22321 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.121352 22321 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.121417 22321 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20250110 03:07:10.129015 21516 tablet.cc:2367] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:10.173336 22328 raft_consensus.cc:491] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.173421 22328 raft_consensus.cc:513] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.173718 22328 leader_election.cc:290] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:10.176370 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250f547c54a84a35b2850d50b897810e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:10.176359 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250f547c54a84a35b2850d50b897810e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.176481 21600 raft_consensus.cc:2463] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:10.176487 21731 raft_consensus.cc:2463] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:10.176677 21274 leader_election.cc:304] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:10.176822 22328 raft_consensus.cc:2798] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.176880 22328 raft_consensus.cc:491] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.176898 22328 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.177547 22328 raft_consensus.cc:513] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.177659 22328 leader_election.cc:290] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:10.177806 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250f547c54a84a35b2850d50b897810e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:10.177836 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250f547c54a84a35b2850d50b897810e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.177871 21600 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.177887 21731 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.178433 21600 raft_consensus.cc:2463] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:10.178458 21731 raft_consensus.cc:2463] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:10.178619 21274 leader_election.cc:304] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:10.178759 22328 raft_consensus.cc:2798] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.178884 22328 raft_consensus.cc:695] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:10.178956 22328 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.179634 21195 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:10.184265 22334 raft_consensus.cc:491] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.184340 22334 raft_consensus.cc:513] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.184543 22334 leader_election.cc:290] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:10.186944 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcde555c3b364cf3b9b582ee3e273f3e" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.186967 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcde555c3b364cf3b9b582ee3e273f3e" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:10.187004 21731 raft_consensus.cc:2463] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.187036 21338 raft_consensus.cc:2463] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.187170 21405 leader_election.cc:304] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.187285 22334 raft_consensus.cc:2798] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.187356 22334 raft_consensus.cc:491] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.187387 22334 raft_consensus.cc:3054] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.187937 22334 raft_consensus.cc:513] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.188055 22334 leader_election.cc:290] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:10.188185 21731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcde555c3b364cf3b9b582ee3e273f3e" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.188266 21731 raft_consensus.cc:3054] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.188251 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcde555c3b364cf3b9b582ee3e273f3e" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:10.188298 21338 raft_consensus.cc:3054] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.188717 21731 raft_consensus.cc:2463] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.188807 21338 raft_consensus.cc:2463] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.188879 21405 leader_election.cc:304] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.188961 22334 raft_consensus.cc:2798] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.189079 22334 raft_consensus.cc:695] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:10.189155 22334 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.189739 21195 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:10.247493 21647 tablet.cc:2367] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:07:10.250643 21385 tablet.cc:2367] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:10.357617 22335 raft_consensus.cc:491] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.357704 22335 raft_consensus.cc:513] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.358003 22335 leader_election.cc:290] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:10.360580 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:10.360678 21338 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:10.360826 21534 leader_election.cc:304] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:10.360952 22335 raft_consensus.cc:2798] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.361019 22335 raft_consensus.cc:491] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.361047 22335 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.361372 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:10.361470 21469 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:10.361707 22335 raft_consensus.cc:513] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.361829 22335 leader_election.cc:290] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:10.361944 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:10.362012 21469 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.362004 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:10.362078 21338 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
W20250110 03:07:10.362520 21778 tablet.cc:2367] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:10.362651 21338 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:10.362653 21469 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:10.362784 21534 leader_election.cc:304] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:10.362891 22335 raft_consensus.cc:2798] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.362955 22335 raft_consensus.cc:695] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:10.363030 22335 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.363744 21195 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } }
W20250110 03:07:10.455202 22350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:10.455348 22350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:10.456527 22350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250110 03:07:10.489224 21469 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.489743 22335 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.490584 21338 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.491391 21600 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.491492 22335 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.491660 22341 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.492288 21731 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.492412 21337 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.492291 21730 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.492632 22345 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.492651 22341 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.493011 22345 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.495666 22388 mvcc.cc:204] Tried to move back new op lower bound from 7112615651293143040 to 7112615650013491200. Current Snapshot: MvccSnapshot[applied={T|T < 7112615651290181632}]
I20250110 03:07:10.496551 22382 mvcc.cc:204] Tried to move back new op lower bound from 7112615651290259456 to 7112615650767601664. Current Snapshot: MvccSnapshot[applied={T|T < 7112615651281137664}]
I20250110 03:07:10.497941 22384 mvcc.cc:204] Tried to move back new op lower bound from 7112615651294609408 to 7112615650055135232. Current Snapshot: MvccSnapshot[applied={T|T < 7112615651294609408}]
I20250110 03:07:10.500211 22392 mvcc.cc:204] Tried to move back new op lower bound from 7112615651294609408 to 7112615650055135232. Current Snapshot: MvccSnapshot[applied={T|T < 7112615651294609408}]
INSERT report
rows total: 2000
time total: 26.041 ms
time per row: 0.0130205 ms
I20250110 03:07:10.509819 21194 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:10.511173 21580 tablet_service.cc:1467] Processing CreateTablet for tablet a6000ace3863451db3fdd8d150cd0944 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.511186 21448 tablet_service.cc:1467] Processing CreateTablet for tablet a6000ace3863451db3fdd8d150cd0944 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.511307 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6000ace3863451db3fdd8d150cd0944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.511305 21580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6000ace3863451db3fdd8d150cd0944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.511355 21579 tablet_service.cc:1467] Processing CreateTablet for tablet 41db68f089644324af518422d2cad2fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.511361 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 49c3c5c20dcc4474b207b9c73904219f (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.511466 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.511507 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.511591 21447 tablet_service.cc:1467] Processing CreateTablet for tablet 49c3c5c20dcc4474b207b9c73904219f (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.511672 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.512573 22322 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:10.512599 21449 tablet_service.cc:1467] Processing CreateTablet for tablet 41db68f089644324af518422d2cad2fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.512692 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.512914 22322 tablet_bootstrap.cc:654] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.512894 21710 tablet_service.cc:1467] Processing CreateTablet for tablet a6000ace3863451db3fdd8d150cd0944 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.512998 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6000ace3863451db3fdd8d150cd0944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.513516 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 49c3c5c20dcc4474b207b9c73904219f (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.513617 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.513739 22327 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.514129 22327 tablet_bootstrap.cc:654] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.514209 22321 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.514223 21711 tablet_service.cc:1467] Processing CreateTablet for tablet 41db68f089644324af518422d2cad2fc (DEFAULT_TABLE table=rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.514681 21711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.514828 22323 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.515159 22322 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:10.515202 22322 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:10.515214 22323 tablet_bootstrap.cc:654] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.515352 22322 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.515415 22322 raft_consensus.cc:383] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.515440 22322 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.515494 22322 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.515599 22322 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.515620 22321 tablet_bootstrap.cc:654] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.515661 22322 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:10.516032 22322 tablet_bootstrap.cc:654] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.516077 22323 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.516117 22323 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.516263 22323 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.516324 22321 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.516331 22323 raft_consensus.cc:383] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.516359 22323 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.516366 22321 ts_tablet_manager.cc:1397] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:10.516405 22323 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.516510 22323 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.516508 22321 raft_consensus.cc:357] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.516561 22321 raft_consensus.cc:383] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.516574 22321 raft_consensus.cc:738] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.516559 22327 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.516602 22327 ts_tablet_manager.cc:1397] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:10.516611 22321 consensus_queue.cc:260] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.516712 22321 ts_tablet_manager.cc:1428] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.516768 22321 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.516749 22327 raft_consensus.cc:357] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.516808 22327 raft_consensus.cc:383] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.516839 22327 raft_consensus.cc:738] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.516885 22327 consensus_queue.cc:260] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.516996 22327 ts_tablet_manager.cc:1428] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.517002 22322 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:10.517043 22322 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:10.517063 22327 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.517163 22321 tablet_bootstrap.cc:654] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.517202 22322 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.517278 22322 raft_consensus.cc:383] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.517308 22322 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.517343 22322 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.517443 22327 tablet_bootstrap.cc:654] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.517448 22322 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:10.517520 22322 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:10.517889 22321 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.517936 22321 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.517948 22322 tablet_bootstrap.cc:654] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.518083 22321 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.518137 22321 raft_consensus.cc:383] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.518157 22321 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.518195 22321 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.518263 22403 raft_consensus.cc:491] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.518296 22321 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.518319 22403 raft_consensus.cc:513] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.518504 22403 leader_election.cc:290] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:10.518878 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db68f089644324af518422d2cad2fc" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
W20250110 03:07:10.519209 21535 leader_election.cc:343] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): Illegal state: must be running to vote when last-logged opid is not known
I20250110 03:07:10.519433 22327 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.519480 22327 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:10.519562 22322 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:10.519606 22322 ts_tablet_manager.cc:1397] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:10.519610 22327 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.519663 22327 raft_consensus.cc:383] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.519685 22327 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.519721 22322 raft_consensus.cc:357] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.519727 22327 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.519766 22322 raft_consensus.cc:383] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.519783 22322 raft_consensus.cc:738] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.519809 22327 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.519817 22322 consensus_queue.cc:260] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.519850 22327 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.519891 22322 ts_tablet_manager.cc:1428] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.520242 22327 tablet_bootstrap.cc:654] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.520828 22327 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.520866 22327 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:10.520984 22327 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.521023 22327 raft_consensus.cc:383] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.521035 22327 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.521065 22327 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.521153 22327 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.521894 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db68f089644324af518422d2cad2fc" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.521972 21730 raft_consensus.cc:2463] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:10.522102 21536 leader_election.cc:304] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb; no voters: ab3ed3d8c2394f78943461e22853c7ac
I20250110 03:07:10.522321 22406 raft_consensus.cc:2798] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.522363 22406 raft_consensus.cc:491] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.522385 22406 raft_consensus.cc:3054] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.522940 22406 raft_consensus.cc:513] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.523061 22406 leader_election.cc:290] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:10.523209 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db68f089644324af518422d2cad2fc" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.523219 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41db68f089644324af518422d2cad2fc" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:10.523262 21730 raft_consensus.cc:3054] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.523281 21469 raft_consensus.cc:3054] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.523736 21469 raft_consensus.cc:2463] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:10.523895 21535 leader_election.cc:304] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:10.523972 22406 raft_consensus.cc:2798] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.524014 22406 raft_consensus.cc:695] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:10.524040 21730 raft_consensus.cc:2463] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:10.524060 22406 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.524554 21194 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:10.606362 22345 raft_consensus.cc:491] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.606467 22345 raft_consensus.cc:513] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.606715 22345 leader_election.cc:290] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:10.606914 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6000ace3863451db3fdd8d150cd0944" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.607023 21730 raft_consensus.cc:2463] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.607200 21405 leader_election.cc:304] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.607299 22345 raft_consensus.cc:2798] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.607368 22345 raft_consensus.cc:491] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.607389 22345 raft_consensus.cc:3054] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.608047 22345 raft_consensus.cc:513] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.608177 22345 leader_election.cc:290] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:10.608361 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6000ace3863451db3fdd8d150cd0944" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.608435 21730 raft_consensus.cc:3054] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.609004 21730 raft_consensus.cc:2463] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.609172 21405 leader_election.cc:304] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.609267 22345 raft_consensus.cc:2798] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.609338 22345 raft_consensus.cc:695] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:10.609412 22345 consensus_queue.cc:237] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:10.609431 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6000ace3863451db3fdd8d150cd0944" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:10.609467 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6000ace3863451db3fdd8d150cd0944" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:10.609519 21600 raft_consensus.cc:2463] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.609972 21194 catalog_manager.cc:5526] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:10.646425 22345 raft_consensus.cc:491] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.646502 22345 raft_consensus.cc:513] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.646641 22345 leader_election.cc:290] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:10.646852 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c3c5c20dcc4474b207b9c73904219f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:10.646905 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c3c5c20dcc4474b207b9c73904219f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:10.646944 21599 raft_consensus.cc:2463] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.646979 21337 raft_consensus.cc:2463] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.647100 21403 leader_election.cc:304] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.647184 22345 raft_consensus.cc:2798] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.647221 22345 raft_consensus.cc:491] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.647243 22345 raft_consensus.cc:3054] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.647735 22345 raft_consensus.cc:513] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.647822 22345 leader_election.cc:290] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:10.647966 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c3c5c20dcc4474b207b9c73904219f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:10.648046 21599 raft_consensus.cc:3054] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.648020 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49c3c5c20dcc4474b207b9c73904219f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:10.648082 21337 raft_consensus.cc:3054] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.648821 21337 raft_consensus.cc:2463] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.648823 21599 raft_consensus.cc:2463] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.648978 21403 leader_election.cc:304] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.649067 22345 raft_consensus.cc:2798] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.649117 22345 raft_consensus.cc:695] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:10.649152 22345 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.649724 21194 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:10.731038 22425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:10.731180 22425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:10.732388 22425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250110 03:07:10.762176 21730 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.762385 21599 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.762452 21337 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.762494 21600 raft_consensus.cc:3054] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.762826 22345 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.763371 21600 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.763442 22406 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.763635 21730 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.763629 21469 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.763829 22334 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.763975 22452 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.764055 22334 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.764056 22406 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 21.244 ms
time per row: 0.010622 ms
I20250110 03:07:10.778533 21189 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:10.779982 21711 tablet_service.cc:1467] Processing CreateTablet for tablet b5064ab85103488f873fffef55fd78fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.779994 21449 tablet_service.cc:1467] Processing CreateTablet for tablet b5064ab85103488f873fffef55fd78fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.780035 21710 tablet_service.cc:1467] Processing CreateTablet for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.780100 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.780100 21711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.780077 21316 tablet_service.cc:1467] Processing CreateTablet for tablet b5064ab85103488f873fffef55fd78fd (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.780145 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.780174 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.780331 21447 tablet_service.cc:1467] Processing CreateTablet for tablet 62aacc8deebd43debd76a2c10815da2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.780422 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.780907 21317 tablet_service.cc:1467] Processing CreateTablet for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.780999 21317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.781029 21709 tablet_service.cc:1467] Processing CreateTablet for tablet 62aacc8deebd43debd76a2c10815da2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.781122 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 62aacc8deebd43debd76a2c10815da2f (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.781157 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 (DEFAULT_TABLE table=rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.781258 21580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.781715 22323 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.781891 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.782121 22323 tablet_bootstrap.cc:654] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.782424 22322 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:10.782622 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.783047 22322 tablet_bootstrap.cc:654] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.783638 22322 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:10.783703 22322 ts_tablet_manager.cc:1397] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.783834 22322 raft_consensus.cc:357] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.783895 22322 raft_consensus.cc:383] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.783921 22322 raft_consensus.cc:738] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.783982 22322 consensus_queue.cc:260] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.784080 22322 ts_tablet_manager.cc:1428] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.784343 22327 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.784855 22327 tablet_bootstrap.cc:654] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.785367 22321 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.785640 22323 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.785671 22323 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20250110 03:07:10.785710 22321 tablet_bootstrap.cc:654] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.785775 22323 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.785815 22323 raft_consensus.cc:383] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.785826 22323 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.785862 22323 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.785959 22323 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.786003 22323 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.786171 22327 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.786216 22327 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:10.786329 22327 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.786370 22327 raft_consensus.cc:383] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.786382 22327 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.786379 22323 tablet_bootstrap.cc:654] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.786402 22321 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.786417 22327 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.786441 22321 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:10.786514 22327 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.786589 22327 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.786621 22321 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.786689 22321 raft_consensus.cc:383] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.786721 22321 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.786768 22321 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.786871 22321 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.786923 22321 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.786999 22327 tablet_bootstrap.cc:654] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.787292 22321 tablet_bootstrap.cc:654] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.787899 22327 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:10.787923 22321 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.787945 22327 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:10.787963 22321 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:10.788094 22327 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.788094 22321 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.788144 22327 raft_consensus.cc:383] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.788144 22321 raft_consensus.cc:383] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.788159 22327 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.788177 22321 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.788194 22327 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.788236 22321 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.788282 22327 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.788342 22321 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.788395 22321 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:10.789062 22321 tablet_bootstrap.cc:654] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.789566 22323 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.789600 22321 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:10.789620 22323 ts_tablet_manager.cc:1397] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250110 03:07:10.789634 22321 ts_tablet_manager.cc:1397] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:10.789777 22321 raft_consensus.cc:357] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.789769 22323 raft_consensus.cc:357] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.789831 22323 raft_consensus.cc:383] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.789831 22321 raft_consensus.cc:383] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.789860 22323 raft_consensus.cc:738] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.789860 22321 raft_consensus.cc:738] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.789903 22321 consensus_queue.cc:260] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.789914 22323 consensus_queue.cc:260] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.789991 22321 ts_tablet_manager.cc:1428] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.789995 22323 ts_tablet_manager.cc:1428] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:10.790053 22323 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.790604 22323 tablet_bootstrap.cc:654] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.791172 22323 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.791210 22323 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:10.791328 22323 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.791371 22323 raft_consensus.cc:383] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:10.791383 22323 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:10.791425 22323 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.791507 22323 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.812116 22406 raft_consensus.cc:491] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.812176 22406 raft_consensus.cc:513] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.812320 22406 leader_election.cc:290] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:10.812474 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fe8e3aabc64a848ffc1e4f21fa1b51" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:10.812515 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fe8e3aabc64a848ffc1e4f21fa1b51" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.812544 21337 raft_consensus.cc:2463] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:10.812570 21730 raft_consensus.cc:2463] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:10.812711 21534 leader_election.cc:304] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:10.812798 22406 raft_consensus.cc:2798] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.812837 22406 raft_consensus.cc:491] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.812852 22406 raft_consensus.cc:3054] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.813346 22406 raft_consensus.cc:513] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.813482 22406 leader_election.cc:290] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:10.813616 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fe8e3aabc64a848ffc1e4f21fa1b51" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:10.813634 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fe8e3aabc64a848ffc1e4f21fa1b51" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.813699 21337 raft_consensus.cc:3054] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.813700 21730 raft_consensus.cc:3054] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.814332 21337 raft_consensus.cc:2463] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:10.814332 21730 raft_consensus.cc:2463] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:10.814493 21534 leader_election.cc:304] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:10.814605 22406 raft_consensus.cc:2798] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.814668 22406 raft_consensus.cc:695] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:10.814699 22406 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:10.815235 21190 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:10.820649 22328 raft_consensus.cc:491] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.820715 22328 raft_consensus.cc:513] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.820874 22328 leader_election.cc:290] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:10.821048 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.821115 21730 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:10.821247 21274 leader_election.cc:304] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:10.821350 22328 raft_consensus.cc:2798] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.821403 22328 raft_consensus.cc:491] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.821446 22328 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.822139 22328 raft_consensus.cc:513] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.822292 22328 leader_election.cc:290] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:10.822467 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.822528 21730 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.823014 21730 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:10.823185 21274 leader_election.cc:304] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:10.823284 22328 raft_consensus.cc:2798] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.823361 22328 raft_consensus.cc:695] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:10.823418 22328 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:10.823966 21190 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:10.824074 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:10.824151 21469 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:10.824129 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:10.824204 21468 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.824877 21468 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:10.860158 22452 raft_consensus.cc:491] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:10.860255 22452 raft_consensus.cc:513] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.860414 22452 leader_election.cc:290] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:10.860563 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5064ab85103488f873fffef55fd78fd" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:10.860639 21730 raft_consensus.cc:2463] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.860634 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5064ab85103488f873fffef55fd78fd" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:10.860713 21337 raft_consensus.cc:2463] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:10.860853 21405 leader_election.cc:304] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.860950 22452 raft_consensus.cc:2798] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:10.860991 22452 raft_consensus.cc:491] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:10.861012 22452 raft_consensus.cc:3054] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.861583 22452 raft_consensus.cc:513] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.861729 22452 leader_election.cc:290] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:10.861955 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5064ab85103488f873fffef55fd78fd" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:10.862030 21337 raft_consensus.cc:3054] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.862032 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5064ab85103488f873fffef55fd78fd" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:10.862107 21730 raft_consensus.cc:3054] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:10.862668 21730 raft_consensus.cc:2463] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.862701 21337 raft_consensus.cc:2463] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:10.862818 21405 leader_election.cc:304] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:10.862920 22452 raft_consensus.cc:2798] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:10.862994 22452 raft_consensus.cc:695] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:10.863044 22452 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:10.863575 21190 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:10.945581 22471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:10.945734 22471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:10.946919 22471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20250110 03:07:10.976493 21337 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.976547 21468 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.976715 21730 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.976698 22345 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.976961 22328 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.977511 21730 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.977643 22345 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.978180 22328 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.978724 21730 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.978929 22403 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:10.979300 21337 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:10.981112 22403 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 25.6716 ms
time per row: 0.0128358 ms
I20250110 03:07:10.995378 21190 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:10.997210 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 87f8afb021a7407089c710f6dd588d03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.997226 21447 tablet_service.cc:1467] Processing CreateTablet for tablet 87f8afb021a7407089c710f6dd588d03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.997274 21710 tablet_service.cc:1467] Processing CreateTablet for tablet ea70c2cc77384698aedb06bee2b639ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.997314 21449 tablet_service.cc:1467] Processing CreateTablet for tablet ecbd6afd612b4e86b94429633ac15082 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.997336 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f8afb021a7407089c710f6dd588d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.997342 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f8afb021a7407089c710f6dd588d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.997364 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.997431 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbd6afd612b4e86b94429633ac15082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.998585 22323 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:10.998615 22327 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:10.998996 22327 tablet_bootstrap.cc:654] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.999029 22323 tablet_bootstrap.cc:654] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:10.998550 21709 tablet_service.cc:1467] Processing CreateTablet for tablet ecbd6afd612b4e86b94429633ac15082 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.999539 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbd6afd612b4e86b94429633ac15082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.999691 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 87f8afb021a7407089c710f6dd588d03 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.999727 21579 tablet_service.cc:1467] Processing CreateTablet for tablet ea70c2cc77384698aedb06bee2b639ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:10.999799 21580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f8afb021a7407089c710f6dd588d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.999894 22323 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:10.999936 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.999939 22323 ts_tablet_manager.cc:1397] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.000094 22323 raft_consensus.cc:357] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.000145 22323 raft_consensus.cc:383] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.000164 22323 raft_consensus.cc:738] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.000202 22323 consensus_queue.cc:260] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.000293 22323 ts_tablet_manager.cc:1428] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:10.999732 21578 tablet_service.cc:1467] Processing CreateTablet for tablet ecbd6afd612b4e86b94429633ac15082 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.000473 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbd6afd612b4e86b94429633ac15082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:10.999373 22321 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:11.000995 22321 tablet_bootstrap.cc:654] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.001340 22322 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.001662 22321 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:11.001709 22321 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.001750 21318 tablet_service.cc:1467] Processing CreateTablet for tablet ea70c2cc77384698aedb06bee2b639ce (DEFAULT_TABLE table=rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.001778 22322 tablet_bootstrap.cc:654] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.001847 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.001878 22321 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.001956 22321 raft_consensus.cc:383] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.001983 22321 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.002039 22321 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.002143 22321 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.002218 22321 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:11.002547 22327 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.002616 22327 ts_tablet_manager.cc:1397] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250110 03:07:11.002765 22327 raft_consensus.cc:357] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.002812 22321 tablet_bootstrap.cc:654] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.002827 22327 raft_consensus.cc:383] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.002851 22327 raft_consensus.cc:738] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.002897 22327 consensus_queue.cc:260] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.003007 22327 ts_tablet_manager.cc:1428] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.003069 22327 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:11.003266 22323 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:11.003464 22327 tablet_bootstrap.cc:654] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.003697 22323 tablet_bootstrap.cc:654] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.003995 22327 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.004006 22322 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.004043 22327 ts_tablet_manager.cc:1397] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.004043 22322 ts_tablet_manager.cc:1397] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:11.004179 22327 raft_consensus.cc:357] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.004177 22322 raft_consensus.cc:357] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.004235 22327 raft_consensus.cc:383] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.004235 22322 raft_consensus.cc:383] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.004254 22322 raft_consensus.cc:738] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.004254 22327 raft_consensus.cc:738] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.004249 22321 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:11.004292 22321 ts_tablet_manager.cc:1397] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.004294 22322 consensus_queue.cc:260] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.004302 22327 consensus_queue.cc:260] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.004379 22322 ts_tablet_manager.cc:1428] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.004396 22327 ts_tablet_manager.cc:1428] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.004432 22322 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.004442 22321 raft_consensus.cc:357] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.004492 22321 raft_consensus.cc:383] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.004509 22321 raft_consensus.cc:738] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.004547 22321 consensus_queue.cc:260] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.004649 22321 ts_tablet_manager.cc:1428] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.004930 22322 tablet_bootstrap.cc:654] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.005406 22323 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:11.005450 22323 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.005514 22322 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.005558 22322 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:11.005568 22323 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.005621 22323 raft_consensus.cc:383] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.005640 22323 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.005686 22323 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.005677 22322 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.005726 22322 raft_consensus.cc:383] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.005744 22322 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.005762 22323 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.005787 22322 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.005899 22322 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.005952 22322 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.006291 22322 tablet_bootstrap.cc:654] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.006752 22322 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.006789 22322 ts_tablet_manager.cc:1397] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:11.006892 22322 raft_consensus.cc:357] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.006937 22322 raft_consensus.cc:383] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.006958 22322 raft_consensus.cc:738] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.007012 22322 consensus_queue.cc:260] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.007094 22322 ts_tablet_manager.cc:1428] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.054911 22334 raft_consensus.cc:491] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:11.054971 22334 raft_consensus.cc:513] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.055063 22334 leader_election.cc:290] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:11.055255 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f8afb021a7407089c710f6dd588d03" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:11.055274 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f8afb021a7407089c710f6dd588d03" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:11.055330 21338 raft_consensus.cc:2463] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:11.055352 21599 raft_consensus.cc:2463] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:11.055478 21403 leader_election.cc:304] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:11.055572 22334 raft_consensus.cc:2798] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:11.055613 22334 raft_consensus.cc:491] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:11.055642 22334 raft_consensus.cc:3054] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.056182 22334 raft_consensus.cc:513] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.056277 22334 leader_election.cc:290] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:11.056445 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f8afb021a7407089c710f6dd588d03" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:11.056479 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87f8afb021a7407089c710f6dd588d03" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:11.056507 21338 raft_consensus.cc:3054] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.056532 21599 raft_consensus.cc:3054] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.057018 21338 raft_consensus.cc:2463] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:11.057040 21599 raft_consensus.cc:2463] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:11.057202 21403 leader_election.cc:304] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:11.057284 22334 raft_consensus.cc:2798] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:11.057354 22334 raft_consensus.cc:695] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:11.057400 22334 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.057921 21190 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:11.135567 22406 raft_consensus.cc:491] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:11.135639 22406 raft_consensus.cc:513] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.135766 22406 leader_election.cc:290] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:11.135967 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbd6afd612b4e86b94429633ac15082" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:11.135967 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbd6afd612b4e86b94429633ac15082" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:11.136065 21468 raft_consensus.cc:2463] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:11.136066 21730 raft_consensus.cc:2463] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:11.136241 21536 leader_election.cc:304] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:11.136341 22406 raft_consensus.cc:2798] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:11.136394 22406 raft_consensus.cc:491] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:11.136436 22406 raft_consensus.cc:3054] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.137050 22406 raft_consensus.cc:513] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.137173 22406 leader_election.cc:290] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:11.137301 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbd6afd612b4e86b94429633ac15082" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:11.137313 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbd6afd612b4e86b94429633ac15082" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:11.137377 21468 raft_consensus.cc:3054] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.137377 21730 raft_consensus.cc:3054] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.138041 21468 raft_consensus.cc:2463] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:11.138041 21730 raft_consensus.cc:2463] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:11.138200 21535 leader_election.cc:304] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:11.138325 22406 raft_consensus.cc:2798] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:11.138391 22406 raft_consensus.cc:695] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:11.138442 22406 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.139037 21190 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:11.369556 22337 raft_consensus.cc:491] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:11.369657 22337 raft_consensus.cc:513] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.369961 22337 leader_election.cc:290] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:11.372610 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:11.372596 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:11.372694 21338 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 0.
I20250110 03:07:11.372694 21599 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 0.
I20250110 03:07:11.372870 21665 leader_election.cc:304] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:11.372990 22337 raft_consensus.cc:2798] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:11.373045 22337 raft_consensus.cc:491] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:11.373072 22337 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.373713 22337 raft_consensus.cc:513] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.373824 22337 leader_election.cc:290] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:11.373991 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:11.374006 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:11.374063 21338 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.374060 21599 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.374637 21599 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 1.
I20250110 03:07:11.374657 21338 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 1.
I20250110 03:07:11.374810 21665 leader_election.cc:304] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:11.374905 22337 raft_consensus.cc:2798] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:11.375005 22337 raft_consensus.cc:695] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Becoming Leader. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Running, Role: LEADER
I20250110 03:07:11.375075 22337 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.375674 21190 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:11.519383 22522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:11.519577 22522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:11.521452 22522 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250110 03:07:11.531002 22406 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.532702 22335 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20250110 03:07:11.554397 21599 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.554633 22334 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.555217 21338 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.555456 21337 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.555531 21599 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.556008 22337 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.556802 22452 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.556857 22337 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 24.2291 ms
time per row: 0.0121145 ms
I20250110 03:07:11.572396 21189 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:11.573729 21449 tablet_service.cc:1467] Processing CreateTablet for tablet a790eedb619d47c6a3ba7eba194b70c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.573761 21578 tablet_service.cc:1467] Processing CreateTablet for tablet a790eedb619d47c6a3ba7eba194b70c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.573875 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a790eedb619d47c6a3ba7eba194b70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.573897 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a790eedb619d47c6a3ba7eba194b70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.573956 21709 tablet_service.cc:1467] Processing CreateTablet for tablet 9b16001c451b455f9eda42f3c98a2fdd (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.573963 21710 tablet_service.cc:1467] Processing CreateTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.574060 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b16001c451b455f9eda42f3c98a2fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.574169 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.575234 21447 tablet_service.cc:1467] Processing CreateTablet for tablet 9b16001c451b455f9eda42f3c98a2fdd (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.575328 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b16001c451b455f9eda42f3c98a2fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.575311 21317 tablet_service.cc:1467] Processing CreateTablet for tablet 9b16001c451b455f9eda42f3c98a2fdd (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.575311 21318 tablet_service.cc:1467] Processing CreateTablet for tablet a790eedb619d47c6a3ba7eba194b70c1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.575448 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a790eedb619d47c6a3ba7eba194b70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.575459 22586 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:11.575479 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.575562 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.576004 22586 tablet_bootstrap.cc:654] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.576339 22587 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:11.576664 22590 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:11.576733 22587 tablet_bootstrap.cc:654] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.576799 21317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b16001c451b455f9eda42f3c98a2fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.576987 22590 tablet_bootstrap.cc:654] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.576997 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 (DEFAULT_TABLE table=rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.577090 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.577224 22587 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:11.577277 22587 ts_tablet_manager.cc:1397] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.577406 22587 raft_consensus.cc:357] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.577448 22590 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:11.577459 22587 raft_consensus.cc:383] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.577481 22587 raft_consensus.cc:738] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.577491 22590 ts_tablet_manager.cc:1397] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.577529 22587 consensus_queue.cc:260] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.577622 22587 ts_tablet_manager.cc:1428] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.577610 22590 raft_consensus.cc:357] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.577666 22590 raft_consensus.cc:383] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.577687 22590 raft_consensus.cc:738] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.577710 22587 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:11.577752 22590 consensus_queue.cc:260] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.577841 22590 ts_tablet_manager.cc:1428] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.577900 22590 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:11.578084 22587 tablet_bootstrap.cc:654] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.578310 22590 tablet_bootstrap.cc:654] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.578627 22586 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.578688 22586 ts_tablet_manager.cc:1397] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:11.578799 22588 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.578861 22590 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:11.578838 22586 raft_consensus.cc:357] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.578892 22586 raft_consensus.cc:383] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.578902 22590 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.578915 22586 raft_consensus.cc:738] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.578961 22586 consensus_queue.cc:260] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.579038 22586 ts_tablet_manager.cc:1428] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.579036 22590 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.579097 22590 raft_consensus.cc:383] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.579093 22586 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:11.579118 22590 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.579160 22590 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.579249 22590 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.579293 22590 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:11.579516 22586 tablet_bootstrap.cc:654] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.579558 22588 tablet_bootstrap.cc:654] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.579648 22590 tablet_bootstrap.cc:654] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.580122 22586 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.580171 22586 ts_tablet_manager.cc:1397] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:11.580310 22586 raft_consensus.cc:357] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.580386 22586 raft_consensus.cc:383] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.580416 22586 raft_consensus.cc:738] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.580457 22586 consensus_queue.cc:260] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.580503 22587 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:11.580538 22587 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:11.580556 22586 ts_tablet_manager.cc:1428] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.580688 22587 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.580749 22587 raft_consensus.cc:383] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.580768 22587 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.580788 22588 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.580811 22587 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.580835 22588 ts_tablet_manager.cc:1397] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.580915 22587 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.580967 22588 raft_consensus.cc:357] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.581027 22588 raft_consensus.cc:383] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.581112 22588 raft_consensus.cc:738] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.581159 22588 consensus_queue.cc:260] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.581301 22588 ts_tablet_manager.cc:1428] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.581362 22588 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.581414 22590 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:11.581458 22590 ts_tablet_manager.cc:1397] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.581610 22590 raft_consensus.cc:357] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.581672 22590 raft_consensus.cc:383] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.581724 22590 raft_consensus.cc:738] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.581766 22588 tablet_bootstrap.cc:654] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.581779 22590 consensus_queue.cc:260] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.581905 22590 ts_tablet_manager.cc:1428] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:11.582281 22588 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.582326 22588 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.582450 22588 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.582506 22588 raft_consensus.cc:383] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.582530 22588 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.582608 22588 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.582690 22588 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.597280 22579 raft_consensus.cc:491] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:11.597335 22579 raft_consensus.cc:513] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.597501 22579 leader_election.cc:290] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:11.597652 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b16001c451b455f9eda42f3c98a2fdd" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:11.597726 21337 raft_consensus.cc:2463] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 0.
I20250110 03:07:11.597867 21665 leader_election.cc:304] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:11.597954 22579 raft_consensus.cc:2798] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:11.598006 22579 raft_consensus.cc:491] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:11.598029 22579 raft_consensus.cc:3054] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.598594 22579 raft_consensus.cc:513] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.598713 22579 leader_election.cc:290] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:11.598841 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b16001c451b455f9eda42f3c98a2fdd" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:11.598907 21337 raft_consensus.cc:3054] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.599357 21337 raft_consensus.cc:2463] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 1.
I20250110 03:07:11.599545 21665 leader_election.cc:304] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:11.599669 22579 raft_consensus.cc:2798] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:11.599730 22579 raft_consensus.cc:695] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Becoming Leader. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Running, Role: LEADER
I20250110 03:07:11.599771 22579 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.600279 21193 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:11.600375 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b16001c451b455f9eda42f3c98a2fdd" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:11.600379 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b16001c451b455f9eda42f3c98a2fdd" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:11.600440 21468 raft_consensus.cc:2463] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 0.
I20250110 03:07:11.623009 22335 raft_consensus.cc:491] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:11.623075 22335 raft_consensus.cc:513] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.623186 22335 leader_election.cc:290] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:11.623344 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:11.623345 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:11.623416 21730 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:11.623406 21337 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:11.623545 21536 leader_election.cc:304] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:11.623637 22335 raft_consensus.cc:2798] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:11.623684 22335 raft_consensus.cc:491] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:11.623724 22335 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.624238 22335 raft_consensus.cc:513] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.624354 22335 leader_election.cc:290] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:11.624490 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:11.624500 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:11.624554 21337 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.624552 21730 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.625015 21730 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:11.625013 21337 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:11.625157 21534 leader_election.cc:304] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:11.625238 22335 raft_consensus.cc:2798] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:11.625296 22335 raft_consensus.cc:695] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:11.625334 22335 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:11.625895 21193 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:11.769477 22452 raft_consensus.cc:491] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:11.769573 22452 raft_consensus.cc:513] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.769709 22452 leader_election.cc:290] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:11.769932 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a790eedb619d47c6a3ba7eba194b70c1" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:11.769927 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a790eedb619d47c6a3ba7eba194b70c1" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:11.770020 21599 raft_consensus.cc:2463] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:11.770020 21337 raft_consensus.cc:2463] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:11.770213 21403 leader_election.cc:304] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:11.770299 22452 raft_consensus.cc:2798] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:11.770341 22452 raft_consensus.cc:491] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:11.770363 22452 raft_consensus.cc:3054] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.770956 22452 raft_consensus.cc:513] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.771054 22452 leader_election.cc:290] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:11.771225 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a790eedb619d47c6a3ba7eba194b70c1" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:11.771247 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a790eedb619d47c6a3ba7eba194b70c1" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:11.771299 21337 raft_consensus.cc:3054] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.771306 21599 raft_consensus.cc:3054] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.771944 21337 raft_consensus.cc:2463] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:11.771944 21599 raft_consensus.cc:2463] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:11.772130 21403 leader_election.cc:304] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:11.772228 22452 raft_consensus.cc:2798] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:11.772292 22452 raft_consensus.cc:695] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:11.772346 22452 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.772991 21193 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:11.880365 22602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:11.880546 22602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:11.881769 22602 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20250110 03:07:11.911811 21337 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.912583 21599 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.912734 21337 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.912990 22579 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.913703 22334 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.913839 22334 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.914038 21469 raft_consensus.cc:3054] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:11.914777 21338 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.914795 21469 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.914978 22579 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.916169 22335 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:11.916473 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:11.916811 22335 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 24.3041 ms
time per row: 0.0121521 ms
I20250110 03:07:11.931010 21193 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:11.932412 21578 tablet_service.cc:1467] Processing CreateTablet for tablet abfade13c24b4cebaf1ec204947154c0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.932466 21449 tablet_service.cc:1467] Processing CreateTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.932539 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfade13c24b4cebaf1ec204947154c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.932531 21448 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1a954889a9452f8a8ee55aa0a6f26e (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.932583 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.932575 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1a954889a9452f8a8ee55aa0a6f26e (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.932718 21580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1a954889a9452f8a8ee55aa0a6f26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.932741 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1a954889a9452f8a8ee55aa0a6f26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.933826 21710 tablet_service.cc:1467] Processing CreateTablet for tablet abfade13c24b4cebaf1ec204947154c0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.933846 22588 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.933945 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfade13c24b4cebaf1ec204947154c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.934201 22588 tablet_bootstrap.cc:654] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.934471 22586 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:11.934782 22586 tablet_bootstrap.cc:654] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.934896 22587 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:11.934902 21447 tablet_service.cc:1467] Processing CreateTablet for tablet abfade13c24b4cebaf1ec204947154c0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.934990 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfade13c24b4cebaf1ec204947154c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.935098 21317 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1a954889a9452f8a8ee55aa0a6f26e (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.935207 21317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1a954889a9452f8a8ee55aa0a6f26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.935267 22587 tablet_bootstrap.cc:654] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.935827 22587 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:11.935882 22587 ts_tablet_manager.cc:1397] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.932547 21579 tablet_service.cc:1467] Processing CreateTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.936021 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.936021 22587 raft_consensus.cc:357] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.936084 22587 raft_consensus.cc:383] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.936107 22587 raft_consensus.cc:738] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.936153 22587 consensus_queue.cc:260] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.936246 22587 ts_tablet_manager.cc:1428] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:11.936420 22590 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:11.936848 22590 tablet_bootstrap.cc:654] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.937179 21710 tablet_service.cc:1467] Processing CreateTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 (DEFAULT_TABLE table=rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:11.937276 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:11.937352 22588 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.937397 22588 ts_tablet_manager.cc:1397] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250110 03:07:11.937491 22586 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.937526 22586 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:11.937547 22588 raft_consensus.cc:357] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.937608 22588 raft_consensus.cc:383] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.937626 22588 raft_consensus.cc:738] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.937675 22588 consensus_queue.cc:260] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.937654 22586 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:11.937714 22586 raft_consensus.cc:383] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.937736 22586 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.937780 22588 ts_tablet_manager.cc:1428] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.937777 22586 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:11.937830 22588 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.937860 22586 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.937914 22586 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:11.938391 22588 tablet_bootstrap.cc:654] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.938376 22586 tablet_bootstrap.cc:654] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.938716 22587 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:11.939163 22587 tablet_bootstrap.cc:654] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.939239 22590 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:11.939285 22590 ts_tablet_manager.cc:1397] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:11.939426 22590 raft_consensus.cc:357] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.939483 22586 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.939500 22590 raft_consensus.cc:383] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.939502 22588 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.939538 22590 raft_consensus.cc:738] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.939540 22586 ts_tablet_manager.cc:1397] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.939548 22588 ts_tablet_manager.cc:1397] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.939587 22590 consensus_queue.cc:260] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.939692 22590 ts_tablet_manager.cc:1428] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:11.939679 22588 raft_consensus.cc:357] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.939688 22586 raft_consensus.cc:357] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.939761 22586 raft_consensus.cc:383] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.939761 22588 raft_consensus.cc:383] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.939790 22586 raft_consensus.cc:738] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.939803 22588 raft_consensus.cc:738] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.939839 22586 consensus_queue.cc:260] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:11.939848 22588 consensus_queue.cc:260] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.939944 22588 ts_tablet_manager.cc:1428] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.939945 22586 ts_tablet_manager.cc:1428] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.940002 22586 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:11.940021 22588 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:11.940357 22588 tablet_bootstrap.cc:654] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.940506 22586 tablet_bootstrap.cc:654] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:11.941051 22586 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:11.941098 22586 ts_tablet_manager.cc:1397] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:11.941233 22586 raft_consensus.cc:357] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.941295 22586 raft_consensus.cc:383] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.941319 22586 raft_consensus.cc:738] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.941363 22586 consensus_queue.cc:260] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:11.941411 22587 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:11.941455 22587 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:11.941469 22586 ts_tablet_manager.cc:1428] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.941488 22588 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:11.941524 22588 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:11.941592 22587 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:11.941684 22587 raft_consensus.cc:383] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.941648 22588 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:11.941716 22587 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.941717 22588 raft_consensus.cc:383] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:11.941743 22588 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:11.941767 22587 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:11.941787 22588 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:11.941869 22588 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:11.941864 22587 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:12.014328 22561 raft_consensus.cc:491] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.014410 22561 raft_consensus.cc:513] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:12.014539 22561 leader_election.cc:290] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.014739 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e7baec841a4b7e9cb933d786ec2552" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.014747 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e7baec841a4b7e9cb933d786ec2552" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:12.014818 21469 raft_consensus.cc:2463] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:12.014825 21730 raft_consensus.cc:2463] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:12.014946 21535 leader_election.cc:304] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.015046 22561 raft_consensus.cc:2798] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.015097 22561 raft_consensus.cc:491] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.015120 22561 raft_consensus.cc:3054] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.015743 22561 raft_consensus.cc:513] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:12.015865 22561 leader_election.cc:290] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.015990 21469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e7baec841a4b7e9cb933d786ec2552" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:12.016016 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e7baec841a4b7e9cb933d786ec2552" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.016067 21469 raft_consensus.cc:3054] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.016074 21730 raft_consensus.cc:3054] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.016732 21730 raft_consensus.cc:2463] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:12.016736 21469 raft_consensus.cc:2463] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:12.016935 21536 leader_election.cc:304] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.017011 22561 raft_consensus.cc:2798] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.017076 22561 raft_consensus.cc:695] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:12.017123 22561 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:12.017625 21189 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:12.088830 22334 raft_consensus.cc:491] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.088922 22334 raft_consensus.cc:513] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.089088 22334 leader_election.cc:290] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.089282 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfade13c24b4cebaf1ec204947154c0" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.089269 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfade13c24b4cebaf1ec204947154c0" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:12.089367 21730 raft_consensus.cc:2463] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:12.089368 21599 raft_consensus.cc:2463] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:12.089541 21405 leader_election.cc:304] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:12.089644 22334 raft_consensus.cc:2798] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.089692 22334 raft_consensus.cc:491] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.089721 22334 raft_consensus.cc:3054] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.090269 22334 raft_consensus.cc:513] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.090385 22334 leader_election.cc:290] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.090528 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfade13c24b4cebaf1ec204947154c0" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:12.090540 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfade13c24b4cebaf1ec204947154c0" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.090626 21599 raft_consensus.cc:3054] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.090626 21730 raft_consensus.cc:3054] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.091107 21730 raft_consensus.cc:2463] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:12.091261 21405 leader_election.cc:304] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:12.091346 22334 raft_consensus.cc:2798] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.091404 22334 raft_consensus.cc:695] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:12.091464 22334 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.091997 21189 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:12.107062 21599 raft_consensus.cc:2463] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:12.226441 22637 raft_consensus.cc:491] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.226554 22637 raft_consensus.cc:513] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.226733 22637 leader_election.cc:290] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:12.226907 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a954889a9452f8a8ee55aa0a6f26e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:12.226938 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a954889a9452f8a8ee55aa0a6f26e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:12.226997 21599 raft_consensus.cc:2463] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:12.227007 21468 raft_consensus.cc:2463] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:12.227151 21272 leader_election.cc:304] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.227319 22637 raft_consensus.cc:2798] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.227366 22637 raft_consensus.cc:491] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.227380 22637 raft_consensus.cc:3054] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.227948 22637 raft_consensus.cc:513] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.228055 22637 leader_election.cc:290] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:12.228209 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a954889a9452f8a8ee55aa0a6f26e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:12.228212 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a954889a9452f8a8ee55aa0a6f26e" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:12.228267 21599 raft_consensus.cc:3054] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.228271 21468 raft_consensus.cc:3054] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.228803 21468 raft_consensus.cc:2463] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.228813 21599 raft_consensus.cc:2463] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.228971 21272 leader_election.cc:304] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.229063 22637 raft_consensus.cc:2798] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.229122 22637 raft_consensus.cc:695] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:12.229167 22637 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.229744 21189 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:12.361115 22650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:12.361256 22650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:12.362444 22650 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20250110 03:07:12.391139 21730 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.391296 21599 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.391615 21730 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.391602 22452 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.391681 21468 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.391835 22452 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.391954 22335 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.392046 22561 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.396878 21599 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.396945 21469 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.397151 22637 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.397259 22504 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.400663 22573 mvcc.cc:204] Tried to move back new op lower bound from 7112615659072278528 to 7112615657846915072. Current Snapshot: MvccSnapshot[applied={T|T < 7112615659072278528 or (T in {7112615659072278528})}]
I20250110 03:07:12.403213 22636 mvcc.cc:204] Tried to move back new op lower bound from 7112615659098116096 to 7112615658411048960. Current Snapshot: MvccSnapshot[applied={T|T < 7112615659098116096 or (T in {7112615659098116096})}]
INSERT report
rows total: 2000
time total: 25.3311 ms
time per row: 0.0126656 ms
I20250110 03:07:12.410723 21187 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:12.412053 21447 tablet_service.cc:1467] Processing CreateTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.412091 21710 tablet_service.cc:1467] Processing CreateTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.412117 21579 tablet_service.cc:1467] Processing CreateTablet for tablet 32609d619a0242d1ba0a851fa0a7812d (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.412178 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.412175 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.412180 21317 tablet_service.cc:1467] Processing CreateTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.412247 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.412298 21317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.412415 21448 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.412496 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.413443 22588 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:12.413442 22586 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:12.413486 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 32609d619a0242d1ba0a851fa0a7812d (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.413564 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.413816 22588 tablet_bootstrap.cc:654] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.413883 22586 tablet_bootstrap.cc:654] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.413869 21709 tablet_service.cc:1467] Processing CreateTablet for tablet 32609d619a0242d1ba0a851fa0a7812d (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.413964 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.414062 21711 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.414132 21711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.414443 22586 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:12.414482 22586 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.414649 22590 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.414650 22586 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.414708 22586 raft_consensus.cc:383] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.414726 22586 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.414772 22586 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.414883 22586 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:12.414953 22586 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:12.414985 22590 tablet_bootstrap.cc:654] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.415331 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.415364 22586 tablet_bootstrap.cc:654] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.415436 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.415565 22590 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.415628 22590 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.415786 22590 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.415863 22590 raft_consensus.cc:383] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.415892 22590 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.415931 22590 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.416028 22590 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:12.416105 22590 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.416540 22590 tablet_bootstrap.cc:654] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.416530 22586 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:12.416579 22586 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:12.416929 22586 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.417009 22586 raft_consensus.cc:383] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.417040 22586 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.417083 22586 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.417152 22588 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:12.417182 22586 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:12.417187 22590 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.417197 22588 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20250110 03:07:12.417232 22590 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.417331 22588 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.417357 22590 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.417409 22590 raft_consensus.cc:383] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.417411 22588 raft_consensus.cc:383] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.417429 22590 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.417439 22588 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.417481 22590 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.417484 22588 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.417567 22590 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.417582 22588 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.417622 22590 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.418017 22590 tablet_bootstrap.cc:654] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.418644 22590 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.418685 22590 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.418828 22590 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.418874 22590 raft_consensus.cc:383] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.418900 22590 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.418951 22590 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.419051 22590 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.421857 22504 raft_consensus.cc:491] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.421918 22504 raft_consensus.cc:513] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.422057 22504 leader_election.cc:290] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:12.422221 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbef1612c8b4e2baa670b1cbe0e8960" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:12.422299 21468 raft_consensus.cc:2463] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
W20250110 03:07:12.422369 21274 leader_election.cc:343] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:12.422452 21273 leader_election.cc:304] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters: 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:12.422533 22504 raft_consensus.cc:2798] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.422585 22504 raft_consensus.cc:491] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.422610 22504 raft_consensus.cc:3054] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.423072 22504 raft_consensus.cc:513] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.423179 22504 leader_election.cc:290] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:12.423323 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbef1612c8b4e2baa670b1cbe0e8960" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:12.423390 21468 raft_consensus.cc:3054] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
W20250110 03:07:12.423439 21274 leader_election.cc:343] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:12.423835 21468 raft_consensus.cc:2463] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.423990 21273 leader_election.cc:304] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters: 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:12.424068 22504 raft_consensus.cc:2798] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.424109 22504 raft_consensus.cc:695] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:12.424135 22504 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.424535 21187 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:12.427398 22587 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.427724 22587 tablet_bootstrap.cc:654] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.428171 22587 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.428222 22587 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.428373 22587 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.428427 22587 raft_consensus.cc:383] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.428472 22587 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.428515 22587 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.428596 22587 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.428653 22587 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.429026 22587 tablet_bootstrap.cc:654] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.429565 22587 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.429603 22587 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.429693 22587 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.429733 22587 raft_consensus.cc:383] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.429745 22587 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.429773 22587 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.429858 22587 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.429906 22587 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.430415 22587 tablet_bootstrap.cc:654] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.430893 22587 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.430928 22587 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:12.431035 22587 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.431079 22587 raft_consensus.cc:383] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.431092 22587 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.431138 22587 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.431212 22587 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.438843 22335 raft_consensus.cc:491] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.438911 22335 raft_consensus.cc:513] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.439038 22335 leader_election.cc:290] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.439179 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.439193 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:12.439257 21730 raft_consensus.cc:2463] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:12.439257 21336 raft_consensus.cc:2463] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:12.439411 21536 leader_election.cc:304] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.439519 22335 raft_consensus.cc:2798] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.439579 22335 raft_consensus.cc:491] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.439594 22335 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.440124 22335 raft_consensus.cc:513] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.440234 22335 leader_election.cc:290] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.440347 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.440348 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:12.440413 21336 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.440434 21730 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.440909 21336 raft_consensus.cc:2463] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:12.440966 21730 raft_consensus.cc:2463] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:12.441085 21534 leader_election.cc:304] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.441183 22335 raft_consensus.cc:2798] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.441239 22335 raft_consensus.cc:695] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:12.441290 22335 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.441780 21187 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:12.512507 22334 raft_consensus.cc:491] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.512580 22334 raft_consensus.cc:513] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.512724 22334 leader_election.cc:290] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:12.512890 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52b50e3c87b47bead3a25f7255d1c42" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.512915 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52b50e3c87b47bead3a25f7255d1c42" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:12.512979 21336 raft_consensus.cc:2463] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:12.512975 21730 raft_consensus.cc:2463] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:12.513103 21403 leader_election.cc:304] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:12.513206 22334 raft_consensus.cc:2798] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.513249 22334 raft_consensus.cc:491] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.513293 22334 raft_consensus.cc:3054] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.513890 22334 raft_consensus.cc:513] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.514008 22334 leader_election.cc:290] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:12.514142 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52b50e3c87b47bead3a25f7255d1c42" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.514170 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d52b50e3c87b47bead3a25f7255d1c42" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:12.514209 21730 raft_consensus.cc:3054] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.514225 21336 raft_consensus.cc:3054] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.514834 21730 raft_consensus.cc:2463] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:12.514895 21336 raft_consensus.cc:2463] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:12.515023 21405 leader_election.cc:304] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:12.515125 22334 raft_consensus.cc:2798] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.515184 22334 raft_consensus.cc:695] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:12.515233 22334 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.515817 21187 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:12.603541 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:12.603677 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:12.604808 22700 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20250110 03:07:12.633301 21730 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.633301 21336 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.633549 22685 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.633852 21731 raft_consensus.cc:3054] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.634757 21731 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.636075 22637 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.636116 21468 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.636351 22637 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.636384 21336 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.636580 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.637490 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.637724 22334 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.638231 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.640918 22684 mvcc.cc:204] Tried to move back new op lower bound from 7112615660075692032 to 7112615659279872000. Current Snapshot: MvccSnapshot[applied={T|T < 7112615660075692032 or (T in {7112615660075692032})}]
I20250110 03:07:12.645820 22564 mvcc.cc:204] Tried to move back new op lower bound from 7112615660066000896 to 7112615659209490432. Current Snapshot: MvccSnapshot[applied={T|T < 7112615660066000896 or (T in {7112615660066000896})}]
INSERT report
rows total: 2000
time total: 26.9845 ms
time per row: 0.0134922 ms
I20250110 03:07:12.655604 21194 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:12.657047 21448 tablet_service.cc:1467] Processing CreateTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.657078 21579 tablet_service.cc:1467] Processing CreateTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.657100 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11143dad6b402fb0d16c48285646c9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.657097 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.657174 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.657202 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.657215 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e11143dad6b402fb0d16c48285646c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.657348 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.658460 22590 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.658460 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11143dad6b402fb0d16c48285646c9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.658491 22586 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:12.658568 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e11143dad6b402fb0d16c48285646c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.658871 22586 tablet_bootstrap.cc:654] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.658900 22590 tablet_bootstrap.cc:654] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.659391 22586 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:12.659440 22586 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.659544 21448 tablet_service.cc:1467] Processing CreateTablet for tablet 122762e845a046f4a84afeeacc01b42f (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.659572 21710 tablet_service.cc:1467] Processing CreateTablet for tablet 122762e845a046f4a84afeeacc01b42f (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.659591 22586 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.659659 22586 raft_consensus.cc:383] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.659690 22586 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.659698 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122762e845a046f4a84afeeacc01b42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.659729 22586 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.659821 22586 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.659824 22588 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:12.659919 21709 tablet_service.cc:1467] Processing CreateTablet for tablet 4e11143dad6b402fb0d16c48285646c9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.659996 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e11143dad6b402fb0d16c48285646c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.660176 22588 tablet_bootstrap.cc:654] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.660228 22590 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.660271 22590 ts_tablet_manager.cc:1397] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:12.660414 22590 raft_consensus.cc:357] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.660478 22590 raft_consensus.cc:383] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.660495 22590 raft_consensus.cc:738] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.660530 22590 consensus_queue.cc:260] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.660604 22590 ts_tablet_manager.cc:1428] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.660657 22590 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.660940 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 122762e845a046f4a84afeeacc01b42f (DEFAULT_TABLE table=rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.661029 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122762e845a046f4a84afeeacc01b42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.661064 22590 tablet_bootstrap.cc:654] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.659663 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122762e845a046f4a84afeeacc01b42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.661463 22587 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.661870 22587 tablet_bootstrap.cc:654] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.662431 22590 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.662441 22588 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:12.662478 22590 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:12.662478 22588 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:12.662616 22586 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:12.662674 22588 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.662674 22590 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.662745 22588 raft_consensus.cc:383] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.662748 22590 raft_consensus.cc:383] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.662773 22590 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.662773 22588 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.662824 22590 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.662823 22588 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.662909 22588 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.662907 22590 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.662971 22588 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:12.663187 22586 tablet_bootstrap.cc:654] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.663359 22588 tablet_bootstrap.cc:654] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.663556 22587 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.663610 22587 ts_tablet_manager.cc:1397] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:12.663758 22587 raft_consensus.cc:357] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.663846 22587 raft_consensus.cc:383] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.663868 22587 raft_consensus.cc:738] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.663906 22587 consensus_queue.cc:260] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.663993 22587 ts_tablet_manager.cc:1428] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:12.664058 22587 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.664417 22586 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:12.664449 22587 tablet_bootstrap.cc:654] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.664464 22586 ts_tablet_manager.cc:1397] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:12.664592 22586 raft_consensus.cc:357] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.664645 22586 raft_consensus.cc:383] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.664669 22586 raft_consensus.cc:738] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.664718 22586 consensus_queue.cc:260] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.664824 22586 ts_tablet_manager.cc:1428] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.665115 22587 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.665164 22587 ts_tablet_manager.cc:1397] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:12.665326 22587 raft_consensus.cc:357] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.665364 22588 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:12.665390 22587 raft_consensus.cc:383] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.665408 22587 raft_consensus.cc:738] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.665406 22588 ts_tablet_manager.cc:1397] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:12.665458 22587 consensus_queue.cc:260] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.665549 22587 ts_tablet_manager.cc:1428] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.665557 22588 raft_consensus.cc:357] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.665611 22588 raft_consensus.cc:383] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.665643 22588 raft_consensus.cc:738] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.665691 22588 consensus_queue.cc:260] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.665795 22588 ts_tablet_manager.cc:1428] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:12.665860 22588 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:12.666234 22588 tablet_bootstrap.cc:654] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.666764 22588 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:12.666805 22588 ts_tablet_manager.cc:1397] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.667007 22588 raft_consensus.cc:357] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.667052 22588 raft_consensus.cc:383] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.667074 22588 raft_consensus.cc:738] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.667119 22588 consensus_queue.cc:260] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.667201 22588 ts_tablet_manager.cc:1428] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.733716 22579 raft_consensus.cc:491] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.733783 22579 raft_consensus.cc:513] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.733902 22579 leader_election.cc:290] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:12.734089 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11143dad6b402fb0d16c48285646c9" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:12.734164 21338 raft_consensus.cc:2463] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 0.
I20250110 03:07:12.734133 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11143dad6b402fb0d16c48285646c9" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:12.734251 21599 raft_consensus.cc:2463] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 0.
I20250110 03:07:12.734308 21665 leader_election.cc:304] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:12.734404 22579 raft_consensus.cc:2798] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.734463 22579 raft_consensus.cc:491] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.734495 22579 raft_consensus.cc:3054] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.735256 22579 raft_consensus.cc:513] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.735404 22579 leader_election.cc:290] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:12.735555 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11143dad6b402fb0d16c48285646c9" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:12.735644 21338 raft_consensus.cc:3054] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.735625 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e11143dad6b402fb0d16c48285646c9" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:12.735693 21599 raft_consensus.cc:3054] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.736346 21338 raft_consensus.cc:2463] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 1.
I20250110 03:07:12.736361 21599 raft_consensus.cc:2463] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 1.
I20250110 03:07:12.736547 21665 leader_election.cc:304] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:12.736634 22579 raft_consensus.cc:2798] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.736686 22579 raft_consensus.cc:695] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Becoming Leader. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Running, Role: LEADER
I20250110 03:07:12.736724 22579 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.737275 21194 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132). New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:12.745051 22334 raft_consensus.cc:491] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.745110 22334 raft_consensus.cc:513] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.745234 22334 leader_election.cc:290] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:12.745360 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122762e845a046f4a84afeeacc01b42f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:12.745388 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122762e845a046f4a84afeeacc01b42f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.745438 21599 raft_consensus.cc:2463] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:12.745451 21730 raft_consensus.cc:2463] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:12.745579 21403 leader_election.cc:304] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.745671 22334 raft_consensus.cc:2798] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.745705 22334 raft_consensus.cc:491] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.745733 22334 raft_consensus.cc:3054] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.746224 22334 raft_consensus.cc:513] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.746343 22334 leader_election.cc:290] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:12.746466 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122762e845a046f4a84afeeacc01b42f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:12.746466 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122762e845a046f4a84afeeacc01b42f" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.746527 21599 raft_consensus.cc:3054] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.746527 21730 raft_consensus.cc:3054] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.747141 21730 raft_consensus.cc:2463] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:12.747141 21599 raft_consensus.cc:2463] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:12.747298 21403 leader_election.cc:304] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.747390 22334 raft_consensus.cc:2798] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.747440 22334 raft_consensus.cc:695] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:12.747493 22334 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.748037 21194 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:12.791882 22335 raft_consensus.cc:491] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.791942 22335 raft_consensus.cc:513] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.792033 22335 leader_election.cc:290] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:12.792196 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fa30056c0a4b30ab8fcaa14645e7a4" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:12.792193 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fa30056c0a4b30ab8fcaa14645e7a4" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:12.792306 21468 raft_consensus.cc:2463] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:12.792306 21338 raft_consensus.cc:2463] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 0.
I20250110 03:07:12.792433 21535 leader_election.cc:304] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.792526 22335 raft_consensus.cc:2798] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.792565 22335 raft_consensus.cc:491] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.792585 22335 raft_consensus.cc:3054] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.793092 22335 raft_consensus.cc:513] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.793200 22335 leader_election.cc:290] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:12.793344 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fa30056c0a4b30ab8fcaa14645e7a4" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:12.793375 21338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41fa30056c0a4b30ab8fcaa14645e7a4" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:12.793411 21468 raft_consensus.cc:3054] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.793427 21338 raft_consensus.cc:3054] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.793921 21468 raft_consensus.cc:2463] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:12.794057 21338 raft_consensus.cc:2463] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 1.
I20250110 03:07:12.794107 21535 leader_election.cc:304] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.794185 22335 raft_consensus.cc:2798] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.794234 22335 raft_consensus.cc:695] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:12.794286 22335 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.794790 21194 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 0 to 1, leader changed from <none> to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:12.880487 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:12.880643 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:12.881848 22746 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20250110 03:07:12.910441 21599 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.910451 21338 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.910813 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.910938 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.912640 21468 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.912817 22561 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.914178 21338 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.914495 22561 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.916249 21600 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.916394 21730 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:12.916591 22685 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:12.917039 22685 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 23.6459 ms
time per row: 0.011823 ms
I20250110 03:07:12.930238 21191 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:12.931659 21316 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc5781db40c4148ba1889c45405efc2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.931694 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 58e33e63f48f4b758c19774888345ee0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.931742 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc5781db40c4148ba1889c45405efc2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.931777 21448 tablet_service.cc:1467] Processing CreateTablet for tablet 5cc5781db40c4148ba1889c45405efc2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.931798 21316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.931835 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.931875 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.931881 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.932690 21447 tablet_service.cc:1467] Processing CreateTablet for tablet 58e33e63f48f4b758c19774888345ee0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.932773 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.933132 22588 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:12.933166 22590 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.933496 22588 tablet_bootstrap.cc:654] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.933521 22590 tablet_bootstrap.cc:654] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.934077 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.934176 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bb3816ff5a4d44a249b8bbabdc7b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.934216 21709 tablet_service.cc:1467] Processing CreateTablet for tablet 58e33e63f48f4b758c19774888345ee0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.934274 22588 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:12.934216 21710 tablet_service.cc:1467] Processing CreateTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.934326 22588 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.934362 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.934453 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bb3816ff5a4d44a249b8bbabdc7b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.934453 22588 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.934506 22588 raft_consensus.cc:383] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.934525 22588 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.934582 22588 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.934708 22588 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.935310 22588 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:12.935544 22587 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.935745 22588 tablet_bootstrap.cc:654] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.935926 22587 tablet_bootstrap.cc:654] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.935976 22586 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:12.936266 22588 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:12.936316 22588 ts_tablet_manager.cc:1397] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.936501 22586 tablet_bootstrap.cc:654] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.936569 22588 raft_consensus.cc:357] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.936643 22588 raft_consensus.cc:383] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.936674 22588 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.936713 22588 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.936822 22588 ts_tablet_manager.cc:1428] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:12.937144 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:12.937250 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bb3816ff5a4d44a249b8bbabdc7b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:12.938102 22586 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:12.938150 22586 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:12.938352 22586 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.938401 22586 raft_consensus.cc:383] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.938424 22586 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.938493 22586 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.938648 22586 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:12.938743 22586 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:12.938865 22587 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.938911 22587 ts_tablet_manager.cc:1397] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:12.939064 22587 raft_consensus.cc:357] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.939133 22587 raft_consensus.cc:383] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.939159 22587 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.939186 22586 tablet_bootstrap.cc:654] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.939210 22587 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.939324 22587 ts_tablet_manager.cc:1428] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.939386 22587 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:12.939810 22587 tablet_bootstrap.cc:654] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.939879 22586 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:12.939922 22586 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.940091 22586 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.940160 22586 raft_consensus.cc:383] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.940184 22586 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.940233 22586 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.940286 22590 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.940322 22586 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.940322 22590 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20250110 03:07:12.940485 22590 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.940548 22590 raft_consensus.cc:383] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.940569 22590 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.940614 22590 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:12.940723 22590 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.940776 22590 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.941102 22590 tablet_bootstrap.cc:654] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.942134 22590 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.942175 22590 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:12.942289 22590 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.942343 22590 raft_consensus.cc:383] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.942358 22590 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.942402 22590 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.942493 22590 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.942572 22590 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:12.942941 22590 tablet_bootstrap.cc:654] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:12.943382 22587 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:12.943444 22587 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20250110 03:07:12.943514 22590 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:12.943560 22590 ts_tablet_manager.cc:1397] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:12.943603 22587 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.943678 22587 raft_consensus.cc:383] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.943711 22587 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.943688 22590 raft_consensus.cc:357] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.943745 22590 raft_consensus.cc:383] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:12.943774 22590 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:12.943761 22587 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.943819 22590 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.943851 22587 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:12.943931 22590 ts_tablet_manager.cc:1428] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:12.984287 22637 raft_consensus.cc:491] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.984367 22637 raft_consensus.cc:513] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.984498 22637 leader_election.cc:290] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.984661 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:12.984686 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.984745 21599 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:12.984766 21730 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:12.984921 21272 leader_election.cc:304] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.985006 22637 raft_consensus.cc:2798] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.985045 22637 raft_consensus.cc:491] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.985059 22637 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.985630 22637 raft_consensus.cc:513] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.985733 22637 leader_election.cc:290] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:12.985877 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.985875 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:12.985934 21599 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.985934 21730 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.986536 21599 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.986634 21730 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.986783 21272 leader_election.cc:304] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:12.986869 22637 raft_consensus.cc:2798] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.986933 22637 raft_consensus.cc:695] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:12.986984 22637 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } }
I20250110 03:07:12.987522 21191 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:12.997207 22637 raft_consensus.cc:491] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:12.997256 22637 raft_consensus.cc:513] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.997342 22637 leader_election.cc:290] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:12.997470 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e33e63f48f4b758c19774888345ee0" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:12.997542 21730 raft_consensus.cc:2463] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:12.997517 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e33e63f48f4b758c19774888345ee0" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:12.997588 21468 raft_consensus.cc:2463] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:12.997689 21274 leader_election.cc:304] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:12.997781 22637 raft_consensus.cc:2798] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:12.997818 22637 raft_consensus.cc:491] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:12.997848 22637 raft_consensus.cc:3054] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.998306 22637 raft_consensus.cc:513] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.998409 22637 leader_election.cc:290] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:12.998571 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e33e63f48f4b758c19774888345ee0" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:12.998634 21730 raft_consensus.cc:3054] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.998586 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e33e63f48f4b758c19774888345ee0" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:12.998661 21468 raft_consensus.cc:3054] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:12.999095 21730 raft_consensus.cc:2463] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.999176 21468 raft_consensus.cc:2463] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:12.999257 21274 leader_election.cc:304] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422; no voters:
I20250110 03:07:12.999363 22637 raft_consensus.cc:2798] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:12.999421 22637 raft_consensus.cc:695] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:12.999467 22637 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:12.999979 21191 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:13.238298 22637 raft_consensus.cc:491] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:13.238387 22637 raft_consensus.cc:513] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.238574 22637 leader_election.cc:290] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:13.238720 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac" is_pre_election: true
I20250110 03:07:13.238747 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:13.238823 21468 raft_consensus.cc:2463] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:13.238852 21600 raft_consensus.cc:2463] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 0.
I20250110 03:07:13.238960 21273 leader_election.cc:304] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:13.239063 22637 raft_consensus.cc:2798] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:13.239089 22637 raft_consensus.cc:491] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:13.239104 22637 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.239650 22637 raft_consensus.cc:513] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.239753 22637 leader_election.cc:290] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Requested vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:13.239892 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:13.239959 21600 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.239940 21468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2" candidate_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:13.239995 21468 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.240554 21600 raft_consensus.cc:2463] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:13.240584 21468 raft_consensus.cc:2463] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2b4fdb2c41d4a7ebf55cfac417c3422 in term 1.
I20250110 03:07:13.240727 21272 leader_election.cc:304] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:13.240861 22637 raft_consensus.cc:2798] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:13.240926 22637 raft_consensus.cc:695] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Leader. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:13.240979 22637 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.241503 21191 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:13.363816 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:13.363965 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:13.365098 22798 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20250110 03:07:13.394980 21729 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.395068 21599 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.394840 21600 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.395242 22825 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.395332 22637 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.395454 22824 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.394502 21730 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.397281 21468 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.397432 21469 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.399186 22824 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.399261 22637 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.399297 22504 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 24.8139 ms
time per row: 0.0124069 ms
I20250110 03:07:13.413393 21194 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:13.414914 21578 tablet_service.cc:1467] Processing CreateTablet for tablet 80152844286545c68c7656ef9bbe8604 (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.414975 21579 tablet_service.cc:1467] Processing CreateTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.415042 21578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80152844286545c68c7656ef9bbe8604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.415028 21318 tablet_service.cc:1467] Processing CreateTablet for tablet 3b266cc7c4f341f785152932febcb79b (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.415095 21579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.415145 21318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.415114 21709 tablet_service.cc:1467] Processing CreateTablet for tablet 80152844286545c68c7656ef9bbe8604 (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.415225 21709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80152844286545c68c7656ef9bbe8604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.416378 22588 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:13.416373 22590 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:13.416404 21710 tablet_service.cc:1467] Processing CreateTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.416491 21710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.416695 22587 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:13.417016 22590 tablet_bootstrap.cc:654] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.417021 22588 tablet_bootstrap.cc:654] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.417081 22587 tablet_bootstrap.cc:654] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.414975 21580 tablet_service.cc:1467] Processing CreateTablet for tablet 3b266cc7c4f341f785152932febcb79b (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.417465 21580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.417644 22587 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:13.417685 22587 ts_tablet_manager.cc:1397] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:13.417809 22587 raft_consensus.cc:357] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.417862 22587 raft_consensus.cc:383] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.417882 22587 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.417929 22587 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.418023 22587 ts_tablet_manager.cc:1428] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.418078 22587 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:13.418435 22590 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422: No bootstrap required, opened a new log
I20250110 03:07:13.418486 22590 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:13.418534 22587 tablet_bootstrap.cc:654] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.418668 22590 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.418736 22590 raft_consensus.cc:383] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.418762 22590 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.418805 22590 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.418848 21449 tablet_service.cc:1467] Processing CreateTablet for tablet 3b266cc7c4f341f785152932febcb79b (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.418893 21447 tablet_service.cc:1467] Processing CreateTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.418912 22590 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.418972 21449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.419173 21447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.420238 22586 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:13.418845 21448 tablet_service.cc:1467] Processing CreateTablet for tablet 80152844286545c68c7656ef9bbe8604 (DEFAULT_TABLE table=rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:13.420464 21448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80152844286545c68c7656ef9bbe8604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:13.420635 22586 tablet_bootstrap.cc:654] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.421030 22587 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: No bootstrap required, opened a new log
I20250110 03:07:13.421082 22587 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20250110 03:07:13.421172 22586 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:13.421219 22586 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:13.421236 22587 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.421317 22587 raft_consensus.cc:383] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.421350 22586 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.421396 22587 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.421437 22586 raft_consensus.cc:383] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.421463 22586 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.421453 22587 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.421500 22586 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.421563 22587 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.421603 22586 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.421685 22586 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:13.422353 22588 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:13.422418 22588 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20250110 03:07:13.422523 22588 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.422616 22588 raft_consensus.cc:383] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.422649 22588 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.422681 22588 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.422794 22588 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.422858 22588 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:13.423240 22588 tablet_bootstrap.cc:654] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.423631 22586 tablet_bootstrap.cc:654] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.424124 22588 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:13.424154 22586 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:13.424168 22588 ts_tablet_manager.cc:1397] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:13.424188 22586 ts_tablet_manager.cc:1397] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:13.424294 22588 raft_consensus.cc:357] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.424351 22588 raft_consensus.cc:383] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.424386 22588 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.424348 22586 raft_consensus.cc:357] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.424428 22586 raft_consensus.cc:383] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.424432 22588 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.424458 22586 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.424501 22586 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.424525 22588 ts_tablet_manager.cc:1428] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.424576 22588 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:13.424593 22586 ts_tablet_manager.cc:1428] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.424680 22586 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:13.424944 22588 tablet_bootstrap.cc:654] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.425056 22586 tablet_bootstrap.cc:654] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:13.425616 22586 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: No bootstrap required, opened a new log
I20250110 03:07:13.425642 22588 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: No bootstrap required, opened a new log
I20250110 03:07:13.425661 22586 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:13.425681 22588 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:13.425781 22586 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.425806 22588 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.425846 22586 raft_consensus.cc:383] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.425856 22588 raft_consensus.cc:383] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:13.425870 22588 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.425869 22586 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: FOLLOWER
I20250110 03:07:13.425911 22588 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.425915 22586 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.425997 22588 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.426028 22586 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.478976 22334 raft_consensus.cc:491] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:13.479053 22334 raft_consensus.cc:513] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.479187 22334 leader_election.cc:290] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:13.479362 21729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:13.479370 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:13.479440 21600 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:13.479446 21729 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:13.479585 21403 leader_election.cc:304] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:13.479696 22334 raft_consensus.cc:2798] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:13.479748 22334 raft_consensus.cc:491] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:13.479776 22334 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.480326 22334 raft_consensus.cc:513] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.480455 22334 leader_election.cc:290] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:13.480614 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:13.480619 21729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:13.480688 21729 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.480685 21600 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.481194 21600 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:13.481244 21729 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:13.481385 21403 leader_election.cc:304] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:13.481493 22334 raft_consensus.cc:2798] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:13.481560 22334 raft_consensus.cc:695] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:13.481617 22334 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } }
I20250110 03:07:13.482146 21191 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:13.498977 22334 raft_consensus.cc:491] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:13.499032 22334 raft_consensus.cc:513] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.499141 22334 leader_election.cc:290] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:13.499269 21729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" is_pre_election: true
I20250110 03:07:13.499284 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:13.499344 21729 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:13.499356 21600 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:13.499506 21405 leader_election.cc:304] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac; no voters:
I20250110 03:07:13.499644 22334 raft_consensus.cc:2798] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:13.499686 22334 raft_consensus.cc:491] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:13.499714 22334 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.500188 22334 raft_consensus.cc:513] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.500276 22334 leader_election.cc:290] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:13.500442 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:13.500468 21729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:13.500509 21600 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.500527 21729 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.501008 21600 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:13.501160 21729 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:13.501223 21403 leader_election.cc:304] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:13.501325 22334 raft_consensus.cc:2798] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:13.501360 22334 raft_consensus.cc:695] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:13.501394 22334 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } }
I20250110 03:07:13.501955 21191 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:13.528082 22452 raft_consensus.cc:491] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:13.528148 22452 raft_consensus.cc:513] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.528267 22452 leader_election.cc:290] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:13.528489 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b266cc7c4f341f785152932febcb79b" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" is_pre_election: true
I20250110 03:07:13.528492 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b266cc7c4f341f785152932febcb79b" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" is_pre_election: true
I20250110 03:07:13.528569 21336 raft_consensus.cc:2463] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:13.528571 21600 raft_consensus.cc:2463] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 0.
I20250110 03:07:13.528713 21403 leader_election.cc:304] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:13.528805 22452 raft_consensus.cc:2798] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:13.528852 22452 raft_consensus.cc:491] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:13.528874 22452 raft_consensus.cc:3054] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.529407 22452 raft_consensus.cc:513] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.529513 22452 leader_election.cc:290] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:13.529686 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b266cc7c4f341f785152932febcb79b" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:13.529713 21600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b266cc7c4f341f785152932febcb79b" candidate_uuid: "ab3ed3d8c2394f78943461e22853c7ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:13.529752 21336 raft_consensus.cc:3054] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.529764 21600 raft_consensus.cc:3054] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:13.530213 21600 raft_consensus.cc:2463] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:13.530303 21336 raft_consensus.cc:2463] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab3ed3d8c2394f78943461e22853c7ac in term 1.
I20250110 03:07:13.530380 21403 leader_election.cc:304] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:13.530474 22452 raft_consensus.cc:2798] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:13.530524 22452 raft_consensus.cc:695] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Leader. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:13.530578 22452 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } }
I20250110 03:07:13.531101 21191 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: term changed from 0 to 1, leader changed from <none> to ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:13.607017 22848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:13.607149 22848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:13.608307 22848 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20250110 03:07:13.640060 21729 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.640233 21600 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.640444 22874 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.640574 21336 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.640616 22873 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.640677 21599 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.640856 22452 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.640885 22873 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.642722 21600 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.643136 21729 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:13.643338 22873 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:13.643458 22452 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 27.1902 ms
time per row: 0.0135951 ms
I20250110 03:07:13.657886 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:42091
--local_ip_for_outbound_sockets=127.14.19.133
--tserver_master_addrs=127.14.19.190:41945
--webserver_port=41347
--webserver_interface=127.14.19.133
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:13.732426 22889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:13.732575 22889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:13.732594 22889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:13.733825 22889 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:13.733866 22889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:13.733891 22889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:07:13.735240 22889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:42091
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=41347
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.22889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:13.735425 22889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:13.735635 22889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:13.739507 22897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:13.740382 22894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:13.741451 22895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:13.742233 22889 server_base.cc:1034] running on GCE node
I20250110 03:07:13.742389 22889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:13.742604 22889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:13.743733 22889 hybrid_clock.cc:648] HybridClock initialized: now 1736478433743714 us; error 34 us; skew 500 ppm
I20250110 03:07:13.744956 22889 webserver.cc:458] Webserver started at http://127.14.19.133:41347/ using document root <none> and password file <none>
I20250110 03:07:13.745168 22889 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:13.745239 22889 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:13.746785 22889 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:07:13.747395 22903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.747550 22889 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:13.747628 22889 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e4d83809ce4746d5ad65e8462857fd9c"
format_stamp: "Formatted at 2025-01-10 03:07:05 on dist-test-slave-nrvn"
I20250110 03:07:13.747884 22889 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:13.784039 22889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:13.784237 22889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:13.784340 22889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:13.784518 22889 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:13.784786 22889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:13.784814 22889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.784845 22889 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:13.784893 22889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.790194 22889 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:42091
I20250110 03:07:13.790258 23016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:42091 every 8 connection(s)
I20250110 03:07:13.790525 22889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:07:13.791940 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 22889
I20250110 03:07:13.792188 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:45351
--local_ip_for_outbound_sockets=127.14.19.134
--tserver_master_addrs=127.14.19.190:41945
--webserver_port=41101
--webserver_interface=127.14.19.134
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:13.795316 23017 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:13.795408 23017 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:13.795574 23017 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:13.795974 21191 ts_manager.cc:194] Re-registered known tserver with Master: e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:13.796532 21191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:60901
W20250110 03:07:13.870431 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:13.870613 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:13.870635 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:13.871830 23020 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:13.871872 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:13.871896 23020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:07:13.873358 23020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:45351
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=41101
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:13.873553 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:13.873733 23020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:13.877753 23025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:13.878723 23026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:13.879838 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:13.880564 23020 server_base.cc:1034] running on GCE node
I20250110 03:07:13.880699 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:13.880867 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:13.881981 23020 hybrid_clock.cc:648] HybridClock initialized: now 1736478433881968 us; error 31 us; skew 500 ppm
I20250110 03:07:13.883033 23020 webserver.cc:458] Webserver started at http://127.14.19.134:41101/ using document root <none> and password file <none>
I20250110 03:07:13.883227 23020 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:13.883301 23020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:13.884339 23020 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:13.884829 23034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.884984 23020 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.885041 23020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "531cf07bcce6435c9e18185c60b2118d"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:13.885285 23020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:13.901607 23020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:13.901813 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:13.901923 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:13.902101 23020 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:13.902359 23020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:13.902391 23020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.902410 23020 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:13.902422 23020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:13.907953 23020 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:45351
I20250110 03:07:13.908015 23147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:45351 every 8 connection(s)
I20250110 03:07:13.908203 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:07:13.912223 23148 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:13.912315 23148 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:13.912495 23148 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:13.912786 21191 ts_manager.cc:194] Re-registered known tserver with Master: 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351)
I20250110 03:07:13.913098 21191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:49759
I20250110 03:07:13.917156 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 23020
I20250110 03:07:13.917404 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.135:39887
--local_ip_for_outbound_sockets=127.14.19.135
--tserver_master_addrs=127.14.19.190:41945
--webserver_port=43769
--webserver_interface=127.14.19.135
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:13.991938 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:13.992107 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:13.992142 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:13.993582 23151 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:13.993625 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:13.993661 23151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.135
I20250110 03:07:13.995029 23151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.135:39887
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.14.19.135
--webserver_port=43769
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.23151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.135
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:13.995260 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:13.995460 23151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:13.999238 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:14.000416 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:14.001113 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:14.002183 23151 server_base.cc:1034] running on GCE node
I20250110 03:07:14.002323 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:14.002481 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:14.003598 23151 hybrid_clock.cc:648] HybridClock initialized: now 1736478434003588 us; error 50 us; skew 500 ppm
I20250110 03:07:14.004703 23151 webserver.cc:458] Webserver started at http://127.14.19.135:43769/ using document root <none> and password file <none>
I20250110 03:07:14.004873 23151 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:14.004935 23151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:14.005941 23151 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:14.006392 23165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.006569 23151 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.006634 23151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b2e7bce763e24b63bc760b0b82b53454"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:14.006857 23151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:14.028573 23151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:14.028733 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:14.028810 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:14.028942 23151 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:14.029150 23151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:14.029175 23151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.029201 23151 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:14.029215 23151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.034593 23151 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.135:39887
I20250110 03:07:14.034636 23278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.135:39887 every 8 connection(s)
I20250110 03:07:14.034945 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20250110 03:07:14.039136 23279 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:14.039206 23279 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:14.039351 23279 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:14.039664 21191 ts_manager.cc:194] Re-registered known tserver with Master: b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887)
I20250110 03:07:14.040009 21191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.135:59169
I20250110 03:07:14.041579 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 23151
I20250110 03:07:14.041798 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.136:42591
--local_ip_for_outbound_sockets=127.14.19.136
--tserver_master_addrs=127.14.19.190:41945
--webserver_port=41375
--webserver_interface=127.14.19.136
--builtin_ntp_servers=127.14.19.148:35525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:14.116441 23282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:14.116587 23282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:14.116605 23282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:14.117805 23282 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:14.117844 23282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:14.117867 23282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.136
I20250110 03:07:14.119283 23282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:35525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.136:42591
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.14.19.136
--webserver_port=41375
--tserver_master_addrs=127.14.19.190:41945
--never_fsync=true
--heap_profile_path=/tmp/kudu.23282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.136
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:14.119467 23282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:14.119673 23282 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:14.123498 23288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:14.124454 23290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:14.125471 23287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:14.126368 23282 server_base.cc:1034] running on GCE node
I20250110 03:07:14.126488 23282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:14.126686 23282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:14.127818 23282 hybrid_clock.cc:648] HybridClock initialized: now 1736478434127804 us; error 28 us; skew 500 ppm
I20250110 03:07:14.129024 23282 webserver.cc:458] Webserver started at http://127.14.19.136:41375/ using document root <none> and password file <none>
I20250110 03:07:14.129195 23282 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:14.129256 23282 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:14.130332 23282 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:14.130833 23296 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.130977 23282 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.131042 23282 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
format_stamp: "Formatted at 2025-01-10 03:07:06 on dist-test-slave-nrvn"
I20250110 03:07:14.131268 23282 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:14.144634 23282 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:14.144814 23282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:14.144904 23282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:14.145052 23282 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:14.145298 23282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:14.145327 23282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.145356 23282 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:14.145382 23282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:14.151330 23282 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.136:42591
I20250110 03:07:14.151372 23409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.136:42591 every 8 connection(s)
I20250110 03:07:14.151634 23282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20250110 03:07:14.155916 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 23282
I20250110 03:07:14.156190 23410 heartbeater.cc:346] Connected to a master server at 127.14.19.190:41945
I20250110 03:07:14.156286 23410 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:14.156450 23410 heartbeater.cc:510] Master 127.14.19.190:41945 requested a full tablet report, sending...
I20250110 03:07:14.156790 21191 ts_manager.cc:194] Re-registered known tserver with Master: d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:14.157197 21191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.136:53171
I20250110 03:07:14.355242 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.356952 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.372490 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.373061 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.374853 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.376183 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.377396 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.384208 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.384258 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.386794 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.397325 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.398083 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.398423 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.403127 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.412485 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.413585 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.416879 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.424456 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.426779 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.427752 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.431000 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.435266 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.439772 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.442338 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.461366 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.463447 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.466846 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.472061 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.472131 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.480753 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.490000 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.490929 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.493316 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.497049 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.501639 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.503382 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.503785 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.512657 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.514017 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.521301 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.561748 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.569346 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.571122 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.573012 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.573724 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.582042 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.588402 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.597589 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.598598 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.603881 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.605020 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.605451 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.606539 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.607278 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.612179 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.619912 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.625833 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.625860 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.625953 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.633801 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.634001 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.634301 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.635809 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.635856 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.637028 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.644145 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.645872 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.649720 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.650166 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.666373 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.668159 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.668177 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.696137 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.697108 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.699704 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.701911 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.708447 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.713114 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.723991 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.729897 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.744905 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.746122 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.753564 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.756258 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.756870 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.756877 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.758430 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.764396 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.789690 21469 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:14.790385 21336 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.790390 21729 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.790745 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.790882 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.793318 22874 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.794735 21193 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.794727 21729 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.796648 21336 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.797554 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.798034 23017 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:14.802389 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.804939 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.805621 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.810164 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.813621 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.818362 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.818841 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.819758 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.822666 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20250110 03:07:14.824249 21404 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52b50e3c87b47bead3a25f7255d1c42. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.826130 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.826758 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.831521 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.833220 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.837669 21469 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:14.837975 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.839699 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.842728 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.843087 21598 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.843279 21336 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.843502 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.843628 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.844733 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.851693 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.854147 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.854657 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.860414 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.863744 22874 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.864588 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.866856 21336 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:14.867291 21403 consensus_peers.cc:487] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87f8afb021a7407089c710f6dd588d03. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.868544 21598 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.869585 21188 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.875396 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.881224 21598 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:14.881866 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.882202 21336 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.882298 22561 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.882834 22561 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.883476 24123 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.883697 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:14.884291 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.884356 21338 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.884653 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.884774 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.887156 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.888037 21190 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.890820 24138 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.892061 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.893260 21190 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.893298 21337 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:14.893966 21469 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.894165 22504 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.894311 21598 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:14.894767 21533 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.895720 22504 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.895870 21338 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.897038 21338 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
W20250110 03:07:14.901603 21272 consensus_peers.cc:487] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a1a954889a9452f8a8ee55aa0a6f26e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.903522 22504 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.904670 21192 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:14.905944 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.906575 21598 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.908941 21469 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.911702 21338 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:14.912905 21469 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.913098 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.913306 23656 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.913445 23656 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.913738 23148 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:14.915445 21467 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:14.916666 21338 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.916831 22884 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.916862 21599 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.917236 22884 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.919361 24168 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
W20250110 03:07:14.920251 21271 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.920456 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.921034 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:14.922425 21190 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:14.924253 21402 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a790eedb619d47c6a3ba7eba194b70c1. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.924310 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:14.927004 24065 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.928825 21338 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.928923 21192 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.930094 21599 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.936457 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.937353 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:14.938045 21599 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:14.938663 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:14.941015 21467 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:14.943044 21729 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.943151 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:14.943449 24065 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.943513 21598 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.943766 21730 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:14.943832 21404 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.944061 22884 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.944444 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.945073 21338 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:14.945163 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.945348 22884 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:14.945534 21403 consensus_peers.cc:487] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122762e845a046f4a84afeeacc01b42f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.945863 21338 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:14.945866 21731 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:14.946043 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:14.946122 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:14.946791 24165 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.946933 21338 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.947963 21731 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.948549 21192 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:14.949167 21534 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.952607 21469 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.953020 22874 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.953310 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.954365 21336 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.954484 21190 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.952662 22884 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.957597 21598 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.958603 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:14.958958 21192 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.961241 21730 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.962525 21729 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.963577 21469 consensus_queue.cc:237] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:14.964183 21598 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.964535 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250110 03:07:14.964577 21404 consensus_peers.cc:487] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6000ace3863451db3fdd8d150cd0944. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.964742 21729 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.964722 22884 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.964906 22884 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.968959 22884 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.970261 21729 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.971612 21192 catalog_manager.cc:5526] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.974174 21598 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:14.975746 21469 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:14.976212 21729 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.976272 21598 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.976362 24065 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:14.976570 21403 consensus_peers.cc:487] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abfade13c24b4cebaf1ec204947154c0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.976751 24065 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.977628 24220 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.978751 21729 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.979818 21190 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.981599 21598 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.981765 21469 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:14.982164 21729 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.982352 24203 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:14.983263 21403 consensus_peers.cc:487] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80152844286545c68c7656ef9bbe8604. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.983528 21598 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.983757 24203 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.986287 21336 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:14.986999 21271 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.987008 21599 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.987134 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:14.987285 24152 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.987373 22504 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.988219 22504 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.989158 21599 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.989965 21190 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:14.991005 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:14.992441 21600 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:14.993496 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.993497 21730 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:14.993686 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:14.993897 21336 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:14.994439 22561 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.994504 21469 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:14.995198 21403 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0760e9b8e3c642f09c3287ca586a2a1d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.995666 21597 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:14.995702 21534 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.995713 21336 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:14.995795 24165 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:14.995872 21728 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.996150 21190 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:14.996672 21271 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:14.996799 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:14.996870 21534 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:14.997093 22884 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:14.997130 21469 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:14.997398 24168 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:14.997733 21336 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.998080 21597 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:14.998265 24168 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:14.998227 22884 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:14.998838 24168 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:14.999486 21729 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.000015 21190 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.000041 21597 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.001286 21195 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.001709 21598 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.003067 21469 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.004364 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.004585 24065 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.009408 24065 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.009666 21597 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.010766 21190 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.011442 21469 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
W20250110 03:07:15.012073 21404 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52b50e3c87b47bead3a25f7255d1c42. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.012297 21336 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.012547 21729 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.012956 24220 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.013089 21403 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52b50e3c87b47bead3a25f7255d1c42. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.013262 24220 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.013723 24220 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.014146 21336 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.014757 21730 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.014991 22504 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.015020 21729 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.015197 21730 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.015192 21336 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.016606 21187 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.017174 21597 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.017635 22504 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.018220 21273 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 250f547c54a84a35b2850d50b897810e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.019191 22504 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.020851 21730 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.022365 21195 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.022513 21597 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.024405 21469 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:15.025166 21403 consensus_peers.cc:487] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80152844286545c68c7656ef9bbe8604. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.025168 21402 consensus_peers.cc:487] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80152844286545c68c7656ef9bbe8604. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.025326 21597 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.025350 21730 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.025501 22874 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.025573 24221 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.026118 24221 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.026321 21730 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.027623 21597 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.027645 21196 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.029924 21730 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.031064 21597 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.031122 21336 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.031337 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.031531 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.039181 22579 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.040611 23279 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:15.040913 21597 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.042297 21336 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.042675 21195 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.047526 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20250110 03:07:15.050508 21664 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.056174 21469 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.056782 21597 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.056833 21336 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:15.057266 21403 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c3c5c20dcc4474b207b9c73904219f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.057364 22874 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.057508 22874 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.058887 22874 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.058998 21597 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.060144 21190 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.060767 21336 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.065004 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.065559 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.078979 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.079720 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.082026 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.088052 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.088400 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.093974 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.095263 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.098639 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.102663 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.102975 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.105780 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.106070 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.114272 21730 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.114902 21336 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.115083 22579 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.115535 21597 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.115721 22579 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.117007 22579 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.118216 21597 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.119235 21190 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:15.120412 21666 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea70c2cc77384698aedb06bee2b639ce. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.120743 21336 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.122715 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.125492 21597 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.126904 21469 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.127123 24165 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.127297 21533 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fa30056c0a4b30ab8fcaa14645e7a4. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.127492 21336 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.127713 24165 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.128410 24165 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.130379 21336 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.130411 21196 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.136518 21336 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:15.137447 21272 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.137621 21271 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.138677 21597 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.139024 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.139128 21469 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.139240 21336 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.139339 22504 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.139515 24138 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.140096 22504 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.140383 24138 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.141098 21730 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.141259 22561 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.141510 21196 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.141619 21730 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:15.142455 21534 consensus_peers.cc:487] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22fe8e3aabc64a848ffc1e4f21fa1b51. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.142843 21336 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.142890 21187 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.144312 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.144377 21597 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.144556 24168 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.145108 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.145715 21469 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:15.145931 21535 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.146009 24123 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.145994 21597 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.146243 21730 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.146695 24122 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.148134 21597 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.148308 24165 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.148883 21336 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:15.149016 21534 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.149159 24165 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:15.149314 21534 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.149760 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.149861 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.149986 22561 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.150130 21336 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.150321 21469 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.151196 21196 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.151430 21336 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.151388 21195 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.151662 24221 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.151980 21533 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.152024 24138 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:15.152227 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.152411 21404 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.152632 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.152732 21730 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.153421 21598 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.153687 24221 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.154417 24221 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.155535 21598 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.156339 21336 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.156795 21196 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index -1 to 4, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.157902 23410 heartbeater.cc:502] Master 127.14.19.190:41945 was elected leader, sending a full tablet report...
I20250110 03:07:15.159227 21336 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.159735 21730 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.159945 24168 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.160509 21467 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.160692 24168 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.160933 24168 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.161774 21730 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.161865 21187 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:15.163357 21272 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.163560 21467 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.164789 21467 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.165421 21598 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.165616 21404 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0760e9b8e3c642f09c3287ca586a2a1d. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.165822 21403 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0760e9b8e3c642f09c3287ca586a2a1d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.165931 24220 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.166159 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.166477 24220 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.166694 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.167248 24220 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.167387 21730 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.167683 24165 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.168095 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.168215 21729 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.169502 21196 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.170997 21467 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.171016 21597 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.171136 24165 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.171577 24165 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.172665 21187 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.172807 21730 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.173100 21467 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.173756 21336 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:15.174531 21272 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.174619 21272 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.174643 21467 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.174912 24168 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.175601 24168 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.176573 21196 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.176589 21467 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.178789 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.179313 21336 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.179494 22884 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.179607 21729 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.179783 21404 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.179927 22884 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.180065 21403 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.180526 22884 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.180693 21729 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.181041 21730 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.181361 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.181800 21336 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.181995 21196 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.182812 21729 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.186920 21729 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:15.187569 21664 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea70c2cc77384698aedb06bee2b639ce. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.187629 21597 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.187671 21336 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.187824 21666 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea70c2cc77384698aedb06bee2b639ce. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.187975 24396 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.188104 24396 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.188844 24397 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.189597 21467 consensus_queue.cc:237] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.189633 21597 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
W20250110 03:07:15.190328 21404 consensus_peers.cc:487] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6000ace3863451db3fdd8d150cd0944. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.190362 21534 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fa30056c0a4b30ab8fcaa14645e7a4. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.190419 21336 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.190868 21598 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.191294 21729 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.191473 24220 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.191957 21597 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.192085 21467 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.192219 24220 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.192278 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.192414 24220 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.192879 21597 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.193140 21338 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.193487 21402 consensus_peers.cc:487] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6000ace3863451db3fdd8d150cd0944. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.193519 21195 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
W20250110 03:07:15.193864 21533 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fa30056c0a4b30ab8fcaa14645e7a4. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.193869 21190 catalog_manager.cc:5526] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.194660 24437 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.194852 24438 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.196256 21338 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.196286 21467 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.197119 21729 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.197259 21190 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.200680 21467 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.200881 21597 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
W20250110 03:07:15.201455 21534 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fa30056c0a4b30ab8fcaa14645e7a4. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.201455 21533 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fa30056c0a4b30ab8fcaa14645e7a4. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.201658 21404 consensus_peers.cc:487] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122762e845a046f4a84afeeacc01b42f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.201747 21729 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.201742 21338 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.201848 21535 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41fa30056c0a4b30ab8fcaa14645e7a4. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.201910 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.202028 21467 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.202073 21403 consensus_peers.cc:487] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122762e845a046f4a84afeeacc01b42f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.202195 21597 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.202378 24203 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.202432 24165 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.202500 24203 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.202595 24165 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.203104 24221 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.203713 21338 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.203864 21187 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.204641 21190 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.206768 21729 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.207074 21597 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.208091 21467 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.210587 21338 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:15.211258 21272 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62aacc8deebd43debd76a2c10815da2f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.212500 21467 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.212699 23656 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.212729 21729 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.212947 23656 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.215374 21469 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.216132 21338 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.216305 21729 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.217470 22504 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
W20250110 03:07:15.218770 21403 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.218879 24065 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.218971 21190 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.219671 21469 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.220531 22884 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.221280 21338 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.221689 21729 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.221844 24065 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.222769 21729 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.222831 21188 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.229216 21469 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.229735 21338 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.229840 21404 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.230036 21729 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.230144 24203 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.230217 21403 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.230278 24203 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.230952 24220 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.231077 21338 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.232039 21729 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.233171 21190 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.238443 21597 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.239177 21469 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.239471 21729 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.239671 22561 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.239796 21534 consensus_peers.cc:487] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecbd6afd612b4e86b94429633ac15082. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.240106 22561 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.240952 21469 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.241540 21730 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.241520 22561 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
W20250110 03:07:15.241658 21402 consensus_peers.cc:487] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122762e845a046f4a84afeeacc01b42f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.242065 21597 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.242401 22874 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250110 03:07:15.242604 21404 consensus_peers.cc:487] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122762e845a046f4a84afeeacc01b42f. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.242841 21403 consensus_peers.cc:487] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122762e845a046f4a84afeeacc01b42f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.242852 21196 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.243340 22874 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.243876 22874 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.244693 21469 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.244921 21730 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.245223 21729 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.246165 21196 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.246527 21597 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.247071 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.249435 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.256108 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.256632 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.257193 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.261497 24631 ts_tablet_manager.cc:927] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.261689 24631 tablet_copy_client.cc:323] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.265209 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.269234 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.269241 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.274142 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.275821 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.278635 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57273)
I20250110 03:07:15.278709 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57273: session id = d0be0af85dee41dd8d01ef4b2af7726f-81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:15.279138 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.279384 21620 tablet_copy_source_session.cc:215] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.280449 24631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.283694 24631 tablet_copy_client.cc:806] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.283746 21597 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.284009 24631 tablet_copy_client.cc:670] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.284025 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20250110 03:07:15.284588 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.284946 21535 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.285051 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.285209 24122 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.285845 24122 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.286002 21469 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.286173 24138 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.286485 21469 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.287144 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.287757 21196 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.288190 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.288766 24631 tablet_copy_client.cc:538] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.289806 24631 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.292245 24631 log.cc:826] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:15.298182 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.299619 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.299970 21338 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.300498 21469 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.300535 21729 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.300786 24168 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.301052 21272 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.301151 23656 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.304718 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.304713 21467 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.305172 23656 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:15.305294 21402 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0760e9b8e3c642f09c3287ca586a2a1d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.305344 21729 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.305343 21404 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0760e9b8e3c642f09c3287ca586a2a1d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.305585 22884 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.306176 22884 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
W20250110 03:07:15.306403 21403 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0760e9b8e3c642f09c3287ca586a2a1d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.306515 21597 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.306725 24065 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.306847 21729 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.307446 21193 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.308084 21597 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.308290 21729 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.310837 21193 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.311163 24631 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.311400 24631 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.311517 24631 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.007s
I20250110 03:07:15.313311 24631 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.313503 24631 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.313575 24631 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.313964 24631 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250110 03:07:15.314200 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-81b3bd116ad54a9fa10d2d0c829bb467 received from {username='slave'} at 127.14.19.136:57273
I20250110 03:07:15.314275 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-81b3bd116ad54a9fa10d2d0c829bb467 on tablet 81b3bd116ad54a9fa10d2d0c829bb467 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.315898 21469 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.321149 21598 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.321869 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.321923 21338 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.322046 23364 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.322197 24437 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.322274 21534 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.322296 24437 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.322420 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.323128 24437 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.323251 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.323251 21338 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.324707 21598 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.325093 21729 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.325372 21469 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.325683 21195 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.326421 24138 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250110 03:07:15.326694 21533 consensus_peers.cc:487] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5d0922741634918830ee2c163cbce24. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.326927 24631 ts_tablet_manager.cc:927] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.327068 24631 tablet_copy_client.cc:323] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.327435 21336 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.327808 24396 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.327899 21467 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.328434 24396 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.330405 21338 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:15.330646 24138 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:15.331451 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet a790eedb619d47c6a3ba7eba194b70c1 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:15.331521 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet a790eedb619d47c6a3ba7eba194b70c1 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:15.332250 21489 tablet_copy_source_session.cc:215] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.333863 24631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a790eedb619d47c6a3ba7eba194b70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.331527 24138 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.333606 21338 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
W20250110 03:07:15.334968 21665 consensus_peers.cc:487] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b16001c451b455f9eda42f3c98a2fdd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.335492 24631 tablet_copy_client.cc:806] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.335584 24631 tablet_copy_client.cc:670] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.336436 21189 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.337256 24396 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.337977 21469 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.337878 21467 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.339360 21336 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.339710 24123 raft_consensus.cc:1059] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:15.340974 24631 tablet_copy_client.cc:538] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.340472 21187 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index -1 to 4, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.341609 24631 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.341066 24123 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.342614 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.343010 24165 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250110 03:07:15.343672 21534 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.345395 23364 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.346278 23363 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.346762 24165 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.347003 24631 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.347070 21336 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.347209 24631 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.347304 24631 ts_tablet_manager.cc:1397] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20250110 03:07:15.347298 24165 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.347407 24631 raft_consensus.cc:357] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.347472 24631 raft_consensus.cc:738] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.347506 24631 consensus_queue.cc:260] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.347993 24138 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.348120 24631 ts_tablet_manager.cc:1428] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:15.348613 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-a790eedb619d47c6a3ba7eba194b70c1 received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:15.348680 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-a790eedb619d47c6a3ba7eba194b70c1 on tablet a790eedb619d47c6a3ba7eba194b70c1 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.349164 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.349601 21188 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.350366 21336 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.352211 23363 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.353407 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.353983 21598 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:15.354497 23363 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.354684 24123 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.354753 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.354512 21336 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.355288 24716 ts_tablet_manager.cc:927] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.355064 22561 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.355459 24716 tablet_copy_client.cc:323] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.355518 24165 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.356462 22561 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.357403 23363 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.357474 21188 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:15.357662 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.358080 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.359092 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.360028 21336 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.361166 21598 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:15.361745 23363 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.361934 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.362943 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.363263 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.363447 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.363796 24138 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.364758 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.364913 21188 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:15.365639 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:15.366107 23363 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.367480 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.370308 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.372515 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.373751 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet abfade13c24b4cebaf1ec204947154c0 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:15.373773 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.373805 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet abfade13c24b4cebaf1ec204947154c0 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:15.374310 21489 tablet_copy_source_session.cc:215] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.375054 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfade13c24b4cebaf1ec204947154c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.377884 24716 tablet_copy_client.cc:806] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.378080 24716 tablet_copy_client.cc:670] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.379735 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49544
W20250110 03:07:15.380414 21181 catalog_manager.cc:4839] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): delete failed for tablet 81b3bd116ad54a9fa10d2d0c829bb467 because the tablet was not found. No further retry: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:15.381158 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.383242 24724 tablet_replica.cc:331] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:15.383379 24724 raft_consensus.cc:2238] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:15.383877 21338 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
W20250110 03:07:15.384721 21272 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.384779 21729 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.384948 21272 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.385000 23656 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.385553 21467 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.385954 21273 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.386015 23656 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.386111 23656 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.386255 21729 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.387341 21467 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.387760 21469 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.383502 24724 raft_consensus.cc:2267] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:15.388440 21598 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.388724 24724 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:15.388967 21187 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.389183 24716 tablet_copy_client.cc:538] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250110 03:07:15.389552 21403 consensus_peers.cc:487] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.389676 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.390411 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.391253 21729 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.390839 24716 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.391450 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.391610 24065 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
W20250110 03:07:15.391844 21402 consensus_peers.cc:487] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abfade13c24b4cebaf1ec204947154c0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.392886 21188 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.392925 21598 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.393640 24716 log.cc:826] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:15.395274 24724 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:15.395395 24724 log.cc:1198] T 81b3bd116ad54a9fa10d2d0c829bb467 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:15.395795 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 81b3bd116ad54a9fa10d2d0c829bb467 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:15.395787 21729 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.402503 24716 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.402809 24716 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.402966 24716 ts_tablet_manager.cc:1397] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20250110 03:07:15.404191 24716 raft_consensus.cc:357] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.404441 24716 raft_consensus.cc:738] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.404570 24716 consensus_queue.cc:260] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.404961 24716 ts_tablet_manager.cc:1428] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 03:07:15.405968 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.406523 21729 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.406796 21535 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.406805 24438 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.406971 21467 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.407263 24438 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.407883 24165 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.409262 21467 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.409647 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-abfade13c24b4cebaf1ec204947154c0 received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:15.409691 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-abfade13c24b4cebaf1ec204947154c0 on tablet abfade13c24b4cebaf1ec204947154c0 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.410291 21192 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:15.412147 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.412597 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.413301 21729 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.413378 21336 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.413659 21403 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c3c5c20dcc4474b207b9c73904219f. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.413731 21403 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c3c5c20dcc4474b207b9c73904219f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.413770 24203 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.414422 21598 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.414914 24755 ts_tablet_manager.cc:927] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.414966 22874 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.415076 22874 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.415495 24755 tablet_copy_client.cc:323] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.416236 21336 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.417886 21598 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.420205 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5a1a954889a9452f8a8ee55aa0a6f26e from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:15.420272 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5a1a954889a9452f8a8ee55aa0a6f26e from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:15.420720 21192 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.420979 21358 tablet_copy_source_session.cc:215] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.422041 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1a954889a9452f8a8ee55aa0a6f26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.425530 24755 tablet_copy_client.cc:806] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.425741 24755 tablet_copy_client.cc:670] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.426470 24755 tablet_copy_client.cc:538] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.427690 24755 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.429579 24755 log.cc:826] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:15.431012 24769 ts_tablet_manager.cc:927] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.432325 24769 tablet_copy_client.cc:323] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.433475 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.434509 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.434927 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.435220 24755 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250110 03:07:15.435247 21535 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.435314 24138 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.435462 24755 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
W20250110 03:07:15.435462 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.435510 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.435570 24755 ts_tablet_manager.cc:1397] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20250110 03:07:15.435595 24138 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.436393 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 250f547c54a84a35b2850d50b897810e from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:36717)
I20250110 03:07:15.436450 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 250f547c54a84a35b2850d50b897810e from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:36717: session id = b2e7bce763e24b63bc760b0b82b53454-250f547c54a84a35b2850d50b897810e
I20250110 03:07:15.436494 24138 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.436725 24755 raft_consensus.cc:357] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.436858 24755 raft_consensus.cc:738] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.436921 21358 tablet_copy_source_session.cc:215] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.436916 24755 consensus_queue.cc:260] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.437145 24755 ts_tablet_manager.cc:1428] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 03:07:15.437378 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-5a1a954889a9452f8a8ee55aa0a6f26e received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:15.437444 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-5a1a954889a9452f8a8ee55aa0a6f26e on tablet 5a1a954889a9452f8a8ee55aa0a6f26e with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.437896 24769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.437944 21187 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.439066 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.438469 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.440652 24769 tablet_copy_client.cc:806] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.441322 24769 tablet_copy_client.cc:670] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.443504 24769 tablet_copy_client.cc:538] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.444003 24716 ts_tablet_manager.cc:927] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.444142 24716 tablet_copy_client.cc:323] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.444275 24769 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.444528 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 87f8afb021a7407089c710f6dd588d03 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:15.444588 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 87f8afb021a7407089c710f6dd588d03 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-87f8afb021a7407089c710f6dd588d03
I20250110 03:07:15.444967 21489 tablet_copy_source_session.cc:215] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.445300 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f8afb021a7407089c710f6dd588d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.446002 24769 log.cc:826] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:15.446825 24716 tablet_copy_client.cc:806] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.446938 24716 tablet_copy_client.cc:670] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.448437 24716 tablet_copy_client.cc:538] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.449169 24716 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.450645 24769 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.450942 24769 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.451085 24769 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250110 03:07:15.452167 24769 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.452342 24769 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.452399 24769 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.452605 24769 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:15.453866 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-250f547c54a84a35b2850d50b897810e received from {username='slave'} at 127.14.19.135:36717
I20250110 03:07:15.453918 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-250f547c54a84a35b2850d50b897810e on tablet 250f547c54a84a35b2850d50b897810e with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.458994 21467 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.459564 21598 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.459563 21336 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.459808 21403 consensus_peers.cc:487] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.459872 21403 consensus_peers.cc:487] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87f8afb021a7407089c710f6dd588d03. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.459977 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.460070 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.460848 24220 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.461057 21336 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.461992 21598 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.463755 21188 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.464545 24716 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.464835 24716 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.464938 24716 ts_tablet_manager.cc:1397] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20250110 03:07:15.465062 24716 raft_consensus.cc:357] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.465153 24716 raft_consensus.cc:738] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.465193 24716 consensus_queue.cc:260] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.465415 24716 ts_tablet_manager.cc:1428] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.465595 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-87f8afb021a7407089c710f6dd588d03 received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:15.465654 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-87f8afb021a7407089c710f6dd588d03 on tablet 87f8afb021a7407089c710f6dd588d03 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.477052 24755 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.477216 24755 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.477433 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:15.477499 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:15.478088 21358 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.478722 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.480177 24755 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.480291 24755 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.481729 24755 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.482049 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.482342 24755 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.482645 21729 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.482834 22884 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.483009 21336 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.483290 21404 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.483434 21402 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.483528 21403 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.483450 22884 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.483619 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.484428 21729 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.485172 21336 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.485441 21193 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.487522 24755 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.487743 24755 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.487834 24755 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250110 03:07:15.488001 24755 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.488067 24755 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.488098 24755 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.488252 24755 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.489450 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.490027 24716 ts_tablet_manager.cc:927] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.490151 24716 tablet_copy_client.cc:323] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.490406 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:15.490454 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.492902 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.493825 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 80152844286545c68c7656ef9bbe8604 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:15.493896 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 80152844286545c68c7656ef9bbe8604 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-80152844286545c68c7656ef9bbe8604
I20250110 03:07:15.494434 21489 tablet_copy_source_session.cc:215] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.494783 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80152844286545c68c7656ef9bbe8604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.496143 24716 tablet_copy_client.cc:806] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.496263 24716 tablet_copy_client.cc:670] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.497349 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.497493 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.501864 24716 tablet_copy_client.cc:538] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.502544 24716 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.502853 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.507562 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.509285 21598 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
W20250110 03:07:15.509934 21534 consensus_peers.cc:487] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecbd6afd612b4e86b94429633ac15082. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.510123 21467 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.510274 24165 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.510411 21729 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.510591 24165 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.511538 24165 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.512853 21467 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.514189 21729 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.515345 21194 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
W20250110 03:07:15.515902 21535 consensus_peers.cc:487] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecbd6afd612b4e86b94429633ac15082. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.516575 24716 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.516817 24716 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.516920 24716 ts_tablet_manager.cc:1397] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250110 03:07:15.517040 24716 raft_consensus.cc:357] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.517100 24716 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.517127 24716 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.517422 24716 ts_tablet_manager.cc:1428] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.517589 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-80152844286545c68c7656ef9bbe8604 received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:15.517650 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-80152844286545c68c7656ef9bbe8604 on tablet 80152844286545c68c7656ef9bbe8604 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.523145 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.528752 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.538388 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.544584 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.544929 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.551889 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.553328 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.558055 24755 ts_tablet_manager.cc:927] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.558182 24755 tablet_copy_client.cc:323] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.560918 24631 ts_tablet_manager.cc:927] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:15.561048 24631 tablet_copy_client.cc:323] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:15.562124 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.566710 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 4e11143dad6b402fb0d16c48285646c9 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:41729)
I20250110 03:07:15.566805 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 4e11143dad6b402fb0d16c48285646c9 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:41729: session id = d0be0af85dee41dd8d01ef4b2af7726f-4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:15.567562 21751 tablet_copy_source_session.cc:215] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.567948 24716 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.568096 24716 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.568687 24631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e11143dad6b402fb0d16c48285646c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.570437 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.570464 24631 tablet_copy_client.cc:806] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.570592 24631 tablet_copy_client.cc:670] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.572719 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet b5064ab85103488f873fffef55fd78fd from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:15.572774 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet b5064ab85103488f873fffef55fd78fd from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-b5064ab85103488f873fffef55fd78fd
I20250110 03:07:15.573076 21729 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.573266 21489 tablet_copy_source_session.cc:215] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.573521 21598 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.573734 24344 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.573941 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.574761 24397 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.575273 24755 tablet_copy_client.cc:806] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.575392 24755 tablet_copy_client.cc:670] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
W20250110 03:07:15.575609 21664 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.575786 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.575940 21336 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.576165 21188 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.576844 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.577639 24344 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.577670 21598 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.578112 21336 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.579138 24769 ts_tablet_manager.cc:927] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.579286 24769 tablet_copy_client.cc:323] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.580351 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250110 03:07:15.582720 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.583848 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 3b266cc7c4f341f785152932febcb79b from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:15.583901 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 3b266cc7c4f341f785152932febcb79b from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-3b266cc7c4f341f785152932febcb79b
I20250110 03:07:15.584340 21489 tablet_copy_source_session.cc:215] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.584461 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.584738 24769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.585863 24769 tablet_copy_client.cc:806] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.585959 24769 tablet_copy_client.cc:670] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.590332 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.590735 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.591787 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.592828 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.593525 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.594657 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.597987 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:15.598768 21533 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.598799 21467 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.598876 21535 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.598986 21729 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.598978 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250110 03:07:15.599040 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.599148 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.599620 24123 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.600914 21189 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.601549 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.601598 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.601696 21467 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.603178 21729 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.609102 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.614475 24977 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.614733 24977 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.614751 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.614957 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57273)
I20250110 03:07:15.615010 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57273: session id = d0be0af85dee41dd8d01ef4b2af7726f-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:15.615157 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.615533 21620 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.615847 24977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.617028 24977 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.617125 24977 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.618798 24631 tablet_copy_client.cc:538] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.619318 24965 ts_tablet_manager.cc:927] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.619503 24631 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.619944 24976 ts_tablet_manager.cc:927] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.620064 24976 tablet_copy_client.cc:323] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.620334 24966 ts_tablet_manager.cc:927] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.620487 24966 tablet_copy_client.cc:323] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.621466 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:54949)
I20250110 03:07:15.621511 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:54949: session id = 531cf07bcce6435c9e18185c60b2118d-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:15.621908 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 80152844286545c68c7656ef9bbe8604 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:15.621966 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 80152844286545c68c7656ef9bbe8604 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-80152844286545c68c7656ef9bbe8604
I20250110 03:07:15.621990 21620 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.622046 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet d52b50e3c87b47bead3a25f7255d1c42 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:15.622095 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet d52b50e3c87b47bead3a25f7255d1c42 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:15.622228 24965 tablet_copy_client.cc:323] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.622292 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.622534 21488 tablet_copy_source_session.cc:215] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.622627 21489 tablet_copy_source_session.cc:215] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.622957 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 62aacc8deebd43debd76a2c10815da2f from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:15.623008 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 62aacc8deebd43debd76a2c10815da2f from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:15.623457 21358 tablet_copy_source_session.cc:215] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.623584 24716 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.623694 24716 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.623764 24977 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.624394 24977 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.625234 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.625934 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.626715 24965 tablet_copy_client.cc:806] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.626842 24965 tablet_copy_client.cc:670] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.627456 24966 tablet_copy_client.cc:806] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.627609 24966 tablet_copy_client.cc:670] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.628698 24977 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.628996 24977 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.628957 21336 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:15.629218 24977 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250110 03:07:15.629338 24977 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.629462 24977 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.629513 24977 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.629745 24977 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.629760 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.629966 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.136:57273
I20250110 03:07:15.630045 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.630261 24726 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.630328 24977 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.630416 24977 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.630867 21467 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.631048 22504 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.631163 22504 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:15.632048 24976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80152844286545c68c7656ef9bbe8604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.632567 21467 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:15.633435 24755 tablet_copy_client.cc:538] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.633406 21189 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.634244 24755 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.634703 21729 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:15.635773 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57367)
I20250110 03:07:15.635828 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57367: session id = d0be0af85dee41dd8d01ef4b2af7726f-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:15.636288 21358 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.636634 24716 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250110 03:07:15.636653 21272 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.636759 21272 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.636853 24977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.637369 24716 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.637977 24977 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.638132 24977 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.641155 24755 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.641367 24716 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.641475 24755 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.641571 24716 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.641657 24755 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250110 03:07:15.641670 24716 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250110 03:07:15.641717 21467 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.641772 24716 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.641849 24716 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.641881 24716 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.642278 24716 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20250110 03:07:15.642345 21403 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.642460 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.134:54949
I20250110 03:07:15.642490 21336 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.642524 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.642683 24065 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.643049 24716 ts_tablet_manager.cc:927] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.643222 24716 tablet_copy_client.cc:323] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
W20250110 03:07:15.643251 21403 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52b50e3c87b47bead3a25f7255d1c42. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.643301 24221 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.643558 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:54949)
I20250110 03:07:15.643608 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:54949: session id = 531cf07bcce6435c9e18185c60b2118d-41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:15.644065 21620 tablet_copy_source_session.cc:215] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.644372 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.644431 21336 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.644587 24991 ts_tablet_manager.cc:927] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.644748 24991 tablet_copy_client.cc:323] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.645485 24716 tablet_copy_client.cc:806] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.645581 24716 tablet_copy_client.cc:670] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
W20250110 03:07:15.645815 21404 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d52b50e3c87b47bead3a25f7255d1c42. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.645963 21729 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.646023 21188 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.646135 24220 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.646517 21729 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.647270 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.647609 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.647727 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet d52b50e3c87b47bead3a25f7255d1c42 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:15.647794 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet d52b50e3c87b47bead3a25f7255d1c42 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:15.648286 21489 tablet_copy_source_session.cc:215] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.648506 24965 tablet_copy_client.cc:538] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.649065 24965 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.650219 24755 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.650327 24755 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.650367 24755 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.650707 24755 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.009s user 0.000s sys 0.000s
I20250110 03:07:15.650904 24716 tablet_copy_client.cc:538] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.651522 24716 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.652174 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-b5064ab85103488f873fffef55fd78fd received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:15.652247 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-b5064ab85103488f873fffef55fd78fd on tablet b5064ab85103488f873fffef55fd78fd with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.653034 24976 tablet_copy_client.cc:806] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.653774 24976 tablet_copy_client.cc:670] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.653926 24964 ts_tablet_manager.cc:927] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.654223 24964 tablet_copy_client.cc:323] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.654724 24991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.655957 24965 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.656242 24991 tablet_copy_client.cc:806] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.656195 24965 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.656365 24965 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20250110 03:07:15.656448 24991 tablet_copy_client.cc:670] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.656504 24965 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.656575 24965 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.656608 24965 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.656915 24631 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.657343 24631 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.657425 24631 ts_tablet_manager.cc:1397] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.004s
I20250110 03:07:15.657528 24631 raft_consensus.cc:357] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.657581 24631 raft_consensus.cc:738] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.657606 24631 consensus_queue.cc:260] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.657936 24631 ts_tablet_manager.cc:1428] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.658125 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-4e11143dad6b402fb0d16c48285646c9 received from {username='slave'} at 127.14.19.136:41729
I20250110 03:07:15.658170 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-4e11143dad6b402fb0d16c48285646c9 on tablet 4e11143dad6b402fb0d16c48285646c9 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.658864 24965 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:15.659458 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-62aacc8deebd43debd76a2c10815da2f received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:15.659510 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-62aacc8deebd43debd76a2c10815da2f on tablet 62aacc8deebd43debd76a2c10815da2f with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.661011 24991 tablet_copy_client.cc:538] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.661726 24991 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.662525 24716 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.662765 24716 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.662871 24716 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250110 03:07:15.663007 24716 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.663106 24716 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.663144 24716 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.663635 24976 tablet_copy_client.cc:538] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.663678 24716 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:15.664000 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-41fa30056c0a4b30ab8fcaa14645e7a4 received from {username='slave'} at 127.14.19.134:54949
I20250110 03:07:15.664052 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-41fa30056c0a4b30ab8fcaa14645e7a4 on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.664206 24965 ts_tablet_manager.cc:927] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.664415 24976 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.664312 24965 tablet_copy_client.cc:323] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.664588 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:15.664642 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 44bb3816ff5a4d44a249b8bbabdc7b04 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:15.664894 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.665086 21358 tablet_copy_source_session.cc:215] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.665374 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bb3816ff5a4d44a249b8bbabdc7b04. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 03:07:15.665597 21403 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c3c5c20dcc4474b207b9c73904219f. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.665684 21402 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c3c5c20dcc4474b207b9c73904219f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.666030 21336 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.666165 21598 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.666349 22874 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.666440 24065 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250110 03:07:15.666618 21403 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49c3c5c20dcc4474b207b9c73904219f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.666656 24965 tablet_copy_client.cc:806] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.666805 24965 tablet_copy_client.cc:670] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.666810 24631 ts_tablet_manager.cc:927] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.667330 24965 tablet_copy_client.cc:538] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.667366 24220 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.667492 21336 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.667641 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.667917 24977 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250110 03:07:15.668480 21533 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.668771 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:15.668865 21535 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.668907 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.668969 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.669042 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.668973 22561 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.669153 21188 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.669471 22561 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.670471 24965 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.670475 24977 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.670621 24966 tablet_copy_client.cc:538] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.670558 22561 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.671458 24966 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.671799 21188 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.673563 21598 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.674871 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.676959 24965 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.677165 24965 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.677256 24965 ts_tablet_manager.cc:1397] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20250110 03:07:15.677364 24965 raft_consensus.cc:357] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.677569 24965 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.677654 24965 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.677868 24965 ts_tablet_manager.cc:1428] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250110 03:07:15.678396 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.678625 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet a6000ace3863451db3fdd8d150cd0944 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:15.678704 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet a6000ace3863451db3fdd8d150cd0944 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:15.679311 21488 tablet_copy_source_session.cc:215] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.679653 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-44bb3816ff5a4d44a249b8bbabdc7b04 received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:15.679708 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-44bb3816ff5a4d44a249b8bbabdc7b04 on tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.679865 24769 tablet_copy_client.cc:538] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.680568 24769 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.681203 21729 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.681846 21336 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.682089 21665 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea70c2cc77384698aedb06bee2b639ce. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.682170 24344 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.682355 24631 tablet_copy_client.cc:323] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
W20250110 03:07:15.682487 21666 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea70c2cc77384698aedb06bee2b639ce. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.682835 21664 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea70c2cc77384698aedb06bee2b639ce. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.683081 24343 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.683167 21598 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.684072 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6000ace3863451db3fdd8d150cd0944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.684092 24343 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.685495 24964 tablet_copy_client.cc:806] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.685603 24964 tablet_copy_client.cc:670] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.684541 21336 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.686208 24966 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.686407 24966 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.686517 24966 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20250110 03:07:15.686772 24716 ts_tablet_manager.cc:927] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.686735 24966 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.686923 24966 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.686947 24716 tablet_copy_client.cc:323] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.687021 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57367)
I20250110 03:07:15.687067 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 44bb3816ff5a4d44a249b8bbabdc7b04 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57367: session id = d0be0af85dee41dd8d01ef4b2af7726f-44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:15.687126 24966 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.687184 24977 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.687361 24977 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.687496 24977 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20250110 03:07:15.687536 21358 tablet_copy_source_session.cc:215] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.687588 24966 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:15.687742 24977 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.687858 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-d52b50e3c87b47bead3a25f7255d1c42 received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:15.687914 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-d52b50e3c87b47bead3a25f7255d1c42 on tablet d52b50e3c87b47bead3a25f7255d1c42 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.687896 24977 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.688035 24977 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.688077 24631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bb3816ff5a4d44a249b8bbabdc7b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.688174 21188 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.688598 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:54949)
I20250110 03:07:15.688656 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:54949: session id = 531cf07bcce6435c9e18185c60b2118d-22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:15.689182 21620 tablet_copy_source_session.cc:215] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.689226 24631 tablet_copy_client.cc:806] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.689334 24631 tablet_copy_client.cc:670] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.689509 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.689685 21598 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.689862 24631 tablet_copy_client.cc:538] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.690436 24631 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.690857 24716 tablet_copy_client.cc:806] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.690878 21467 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.690960 24716 tablet_copy_client.cc:670] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.691597 21336 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.692026 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.692090 22884 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.692036 21404 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.692811 22884 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.692919 24977 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20250110 03:07:15.694026 21336 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.694877 21598 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.695122 24221 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.695199 21188 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.695585 21598 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.695696 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.136:57367
I20250110 03:07:15.695740 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.696439 24631 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.696760 24631 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.696990 24631 ts_tablet_manager.cc:1397] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250110 03:07:15.697096 24631 raft_consensus.cc:357] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.697183 24631 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.697252 24769 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.697249 24631 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.697443 24769 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.697494 24631 ts_tablet_manager.cc:1428] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.697546 24769 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250110 03:07:15.697691 24769 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.697764 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-44bb3816ff5a4d44a249b8bbabdc7b04 received from {username='slave'} at 127.14.19.136:57367
I20250110 03:07:15.697806 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-44bb3816ff5a4d44a249b8bbabdc7b04 on tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.697865 24769 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.697963 24769 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.698155 24769 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:15.698347 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-3b266cc7c4f341f785152932febcb79b received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:15.698415 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-3b266cc7c4f341f785152932febcb79b on tablet 3b266cc7c4f341f785152932febcb79b with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.698885 24769 ts_tablet_manager.cc:927] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:15.698894 24716 tablet_copy_client.cc:538] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.699048 24769 tablet_copy_client.cc:323] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:15.699658 24716 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.699731 21598 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.700323 21336 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.700409 21534 consensus_peers.cc:487] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.700492 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.700731 21533 consensus_peers.cc:487] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22fe8e3aabc64a848ffc1e4f21fa1b51. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.700901 21729 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.701072 24122 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.702126 24991 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.701076 24138 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250110 03:07:15.702368 24991 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.702463 24991 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250110 03:07:15.702481 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet ea70c2cc77384698aedb06bee2b639ce from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:38895)
I20250110 03:07:15.702529 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet ea70c2cc77384698aedb06bee2b639ce from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:38895: session id = b2e7bce763e24b63bc760b0b82b53454-ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:15.702659 24991 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.702639 21190 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.702733 24991 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.702677 21336 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.702764 24991 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.702903 24991 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:15.703037 21751 tablet_copy_source_session.cc:215] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.704300 21729 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.704386 24769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.705528 24769 tablet_copy_client.cc:806] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.705637 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42 received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:15.705655 24769 tablet_copy_client.cc:670] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.705693 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42 on tablet d52b50e3c87b47bead3a25f7255d1c42 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.706189 24964 tablet_copy_client.cc:538] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.706240 24769 tablet_copy_client.cc:538] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.706877 24769 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.707002 24964 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.708036 24991 ts_tablet_manager.cc:927] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.708134 24991 tablet_copy_client.cc:323] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.709009 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.709931 22970 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.710244 24065 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.710999 21336 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.711059 24769 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.711172 24065 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.711251 24769 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.711341 24769 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250110 03:07:15.712080 21729 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.712401 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.713083 24220 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
W20250110 03:07:15.713526 21404 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.714066 24976 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.714119 21729 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.714277 24976 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.714382 24976 ts_tablet_manager.cc:1397] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20250110 03:07:15.714481 24976 raft_consensus.cc:357] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.714581 24976 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.714612 24976 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:15.714629 21402 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.714706 24964 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.714877 24976 ts_tablet_manager.cc:1428] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:15.714915 24964 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.714998 24964 ts_tablet_manager.cc:1397] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250110 03:07:15.715106 24964 raft_consensus.cc:357] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.715185 24964 raft_consensus.cc:738] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.715327 24964 consensus_queue.cc:260] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.715613 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-80152844286545c68c7656ef9bbe8604 received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:15.715655 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-80152844286545c68c7656ef9bbe8604 on tablet 80152844286545c68c7656ef9bbe8604 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.715564 21190 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.716048 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:15.716091 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:15.715466 24964 ts_tablet_manager.cc:1428] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:15.716462 21336 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.716549 21620 tablet_copy_source_session.cc:215] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.717204 24991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.717410 24769 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.717507 24769 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.717542 24769 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.717576 24716 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.717633 24769 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250110 03:07:15.717780 24716 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:15.717885 24716 ts_tablet_manager.cc:1397] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20250110 03:07:15.717995 24716 raft_consensus.cc:357] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.718055 24716 raft_consensus.cc:738] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.718081 24716 consensus_queue.cc:260] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.718160 24716 ts_tablet_manager.cc:1428] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.718377 24991 tablet_copy_client.cc:806] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.718382 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-a6000ace3863451db3fdd8d150cd0944 received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:15.718427 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-a6000ace3863451db3fdd8d150cd0944 on tablet a6000ace3863451db3fdd8d150cd0944 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.718472 24991 tablet_copy_client.cc:670] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.718941 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-ea70c2cc77384698aedb06bee2b639ce received from {username='slave'} at 127.14.19.135:38895
I20250110 03:07:15.718992 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-ea70c2cc77384698aedb06bee2b639ce on tablet ea70c2cc77384698aedb06bee2b639ce with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.719095 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-22fe8e3aabc64a848ffc1e4f21fa1b51 received from {username='slave'} at 127.14.19.134:54949
I20250110 03:07:15.719133 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-22fe8e3aabc64a848ffc1e4f21fa1b51 on tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.719559 24755 ts_tablet_manager.cc:927] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:15.719759 24755 tablet_copy_client.cc:323] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:15.723003 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 9b16001c451b455f9eda42f3c98a2fdd from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:51125)
I20250110 03:07:15.723048 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 9b16001c451b455f9eda42f3c98a2fdd from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:51125: session id = e4d83809ce4746d5ad65e8462857fd9c-9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:15.723577 21751 tablet_copy_source_session.cc:215] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.723644 24991 tablet_copy_client.cc:538] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.723940 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b16001c451b455f9eda42f3c98a2fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.724243 24991 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.725072 21336 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.725464 24755 tablet_copy_client.cc:806] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.725641 24755 tablet_copy_client.cc:670] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.725703 21729 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.725891 24152 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.726125 24203 raft_consensus.cc:1059] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:15.726323 24203 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.726235 22970 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.726709 22971 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.726868 21598 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.726910 21272 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.727008 23656 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.727123 23656 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.727317 23364 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.727962 24152 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:15.728037 21404 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.728421 21402 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.728762 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.729547 21598 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.729986 21190 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.730284 23656 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.730801 21729 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.731074 24221 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.731702 22971 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.731812 23364 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.731875 21336 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.732148 22874 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.732169 24769 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.732263 22874 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.732334 24769 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.732499 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:15.732542 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:15.732972 21620 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.733258 24769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.733258 24221 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.733565 24991 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.733758 24991 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.733847 24991 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20250110 03:07:15.733942 24991 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.734004 24991 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.734027 24991 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.734285 21729 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.734388 24991 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:15.734688 24769 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.734791 24769 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.734933 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-41fa30056c0a4b30ab8fcaa14645e7a4 received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:15.734977 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-41fa30056c0a4b30ab8fcaa14645e7a4 on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.735275 21188 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.736210 21336 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.737138 24755 tablet_copy_client.cc:538] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.737854 24755 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.737885 22971 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.738253 21467 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.738888 23233 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.739131 21598 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.739287 21338 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.739462 24065 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.739558 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:15.740096 21404 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.740124 23233 raft_consensus.cc:867] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:15.740172 23233 raft_consensus.cc:1574] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.740360 23233 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.741156 21336 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.741704 24203 raft_consensus.cc:1022] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:15.741730 21729 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.741890 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.741881 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.742040 21272 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 250f547c54a84a35b2850d50b897810e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.742131 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.742167 23232 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.742363 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.742508 24168 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.742693 21729 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.743630 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
W20250110 03:07:15.744266 21402 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.744527 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.744545 25088 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.744741 25088 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.744963 24755 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.745157 24755 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.745251 24755 ts_tablet_manager.cc:1397] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250110 03:07:15.745352 24755 raft_consensus.cc:357] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.745404 24755 raft_consensus.cc:738] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.745421 24755 consensus_queue.cc:260] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.745488 24755 ts_tablet_manager.cc:1428] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.745774 22969 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.745968 21190 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.746045 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-9b16001c451b455f9eda42f3c98a2fdd received from {username='slave'} at 127.14.19.133:51125
I20250110 03:07:15.746088 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-9b16001c451b455f9eda42f3c98a2fdd on tablet 9b16001c451b455f9eda42f3c98a2fdd with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.747220 22504 raft_consensus.cc:1059] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:15.747421 22504 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
W20250110 03:07:15.747637 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.747232 21336 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.747895 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.747987 23364 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.748136 24152 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.748152 21597 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.748211 21729 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:15.748674 21272 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.748833 23656 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.749305 24152 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.749451 24755 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.749560 24755 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.749509 24152 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.749940 21729 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.750958 23364 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.751051 21190 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.751763 21336 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.752012 21597 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:15.752602 21272 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.752849 23656 raft_consensus.cc:1059] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:15.752971 21598 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.752986 23656 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.753329 24065 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.753445 24065 raft_consensus.cc:1022] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:15.753652 21730 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.753798 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.753924 22884 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.754112 23232 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.754456 23364 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.754655 24726 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.754585 22874 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.754740 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.754781 24726 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.755031 21729 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.755010 22504 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.755069 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.755227 24726 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250110 03:07:15.755896 21272 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 250f547c54a84a35b2850d50b897810e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.755914 24726 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.755992 23232 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.756049 21729 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.757067 24221 raft_consensus.cc:1059] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:15.757133 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.757193 24221 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.757323 23233 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.757468 23656 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.757978 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:15.758034 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:15.758343 21190 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.758450 21598 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.758469 22971 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.758703 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.758917 21620 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.758967 23364 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.759310 21336 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.759649 21597 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:15.759994 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.759994 22884 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:15.760123 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.760255 21338 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.760406 21196 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.760582 21188 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 7 to 8, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.760586 21338 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.760725 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.761144 21729 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.761660 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet b5064ab85103488f873fffef55fd78fd with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:15.761718 24755 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.761862 24755 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.761870 23233 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.761916 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:15.762012 24725 tablet_replica.cc:331] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:15.762040 24203 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.762090 24725 raft_consensus.cc:2238] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:15.762140 24725 pending_rounds.cc:62] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Trying to abort 1 pending ops.
I20250110 03:07:15.762173 24725 pending_rounds.cc:69] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7112615672729444352 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } } }
I20250110 03:07:15.762282 24725 raft_consensus.cc:2911] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20250110 03:07:15.762317 24725 raft_consensus.cc:2883] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:15.762336 24725 raft_consensus.cc:2267] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:15.762912 24203 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.762946 21189 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 5 to 6, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.763006 24976 ts_tablet_manager.cc:927] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.763020 24203 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.763144 24976 tablet_copy_client.cc:323] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.763577 23233 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.763727 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 122762e845a046f4a84afeeacc01b42f from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:15.763772 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 122762e845a046f4a84afeeacc01b42f from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-122762e845a046f4a84afeeacc01b42f
I20250110 03:07:15.764048 21597 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.764271 21489 tablet_copy_source_session.cc:215] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.764880 24976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122762e845a046f4a84afeeacc01b42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.764910 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.764995 21338 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:15.765064 21190 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.765486 23364 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:15.765486 21729 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:15.765648 21597 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.765654 25117 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.766010 23656 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.766703 24976 tablet_copy_client.cc:806] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.766906 24976 tablet_copy_client.cc:670] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.767036 23233 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.767096 21338 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.767241 21729 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.767102 24726 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.768329 24769 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20250110 03:07:15.768450 25095 consensus_queue.cc:1159] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e4d83809ce4746d5ad65e8462857fd9c" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20250110 03:07:15.768991 23364 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.769078 24725 ts_tablet_manager.cc:1905] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:15.769095 24769 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.769323 21188 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.769671 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.770416 24725 ts_tablet_manager.cc:1918] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:15.770473 24725 log.cc:1198] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:15.770814 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 44bb3816ff5a4d44a249b8bbabdc7b04 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:15.771164 21336 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:15.771507 23233 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.771530 21729 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.771665 22504 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.771729 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.772881 24755 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.773195 24769 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.773214 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.773422 24769 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.773413 24152 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.773517 24769 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.002s
I20250110 03:07:15.773535 24755 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.773787 21597 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:15.774186 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:15.774267 24152 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.773618 24769 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.774389 24769 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.774446 24769 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.774566 21597 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:15.774871 24976 tablet_copy_client.cc:538] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.774917 24769 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:15.775094 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:15.775142 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.775511 24152 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.775703 24976 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.776002 21597 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.776528 21729 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.777134 21188 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:15.777503 23233 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.777637 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.778209 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 250f547c54a84a35b2850d50b897810e with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49544
W20250110 03:07:15.778488 21181 catalog_manager.cc:4839] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): delete failed for tablet 250f547c54a84a35b2850d50b897810e because the tablet was not found. No further retry: Not found: Tablet not found: 250f547c54a84a35b2850d50b897810e
I20250110 03:07:15.778892 21336 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.779374 21467 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.779436 21273 consensus_peers.cc:487] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a1a954889a9452f8a8ee55aa0a6f26e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.779533 23656 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.779769 22969 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.779834 21338 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:15.780290 23232 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.780442 23656 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.780596 23656 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.780709 23656 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.780844 21597 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:07:15.781147 21181 catalog_manager.cc:4839] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): delete failed for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 because the tablet was not found. No further retry: Not found: Tablet not found: 44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:15.781445 24965 ts_tablet_manager.cc:927] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.781576 24965 tablet_copy_client.cc:323] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.781806 24769 ts_tablet_manager.cc:927] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.781909 24769 tablet_copy_client.cc:323] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.781900 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 122762e845a046f4a84afeeacc01b42f from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:15.781909 24976 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.781940 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 122762e845a046f4a84afeeacc01b42f from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-122762e845a046f4a84afeeacc01b42f
I20250110 03:07:15.782053 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet a6000ace3863451db3fdd8d150cd0944 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:15.782084 24976 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.782083 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet a6000ace3863451db3fdd8d150cd0944 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:15.782171 24976 ts_tablet_manager.cc:1397] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250110 03:07:15.782204 21467 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.782265 24976 raft_consensus.cc:357] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.782323 24976 raft_consensus.cc:738] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.782356 24976 consensus_queue.cc:260] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.782441 24976 ts_tablet_manager.cc:1428] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.782464 21489 tablet_copy_source_session.cc:215] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.782832 21488 tablet_copy_source_session.cc:215] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.782980 24726 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.783200 21190 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.783525 21598 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.783747 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122762e845a046f4a84afeeacc01b42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.783996 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-122762e845a046f4a84afeeacc01b42f received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:15.784040 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-122762e845a046f4a84afeeacc01b42f on tablet 122762e845a046f4a84afeeacc01b42f with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.784261 24769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6000ace3863451db3fdd8d150cd0944. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.784897 24965 tablet_copy_client.cc:806] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.784998 24965 tablet_copy_client.cc:670] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.785369 24769 tablet_copy_client.cc:806] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.785307 24152 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.785476 24769 tablet_copy_client.cc:670] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.785467 23232 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.785519 24991 ts_tablet_manager.cc:927] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.785688 24991 tablet_copy_client.cc:323] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.785899 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 58e33e63f48f4b758c19774888345ee0 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:36717)
I20250110 03:07:15.785944 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 58e33e63f48f4b758c19774888345ee0 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:36717: session id = b2e7bce763e24b63bc760b0b82b53454-58e33e63f48f4b758c19774888345ee0
I20250110 03:07:15.786490 21358 tablet_copy_source_session.cc:215] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.786676 21190 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:15.785259 21597 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:15.787026 24991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.787034 24726 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:15.787305 21597 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:15.787921 24991 tablet_copy_client.cc:806] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.788020 24991 tablet_copy_client.cc:670] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.788777 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 250f547c54a84a35b2850d50b897810e with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:15.788808 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.790251 24769 tablet_copy_client.cc:538] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.790717 21338 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.790864 24769 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.791091 24755 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.791225 22970 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.791262 24755 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.791343 24755 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20250110 03:07:15.791410 24726 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.791427 24755 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.791482 24755 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.791500 24755 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.791574 24755 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.791950 21467 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.792089 24965 tablet_copy_client.cc:538] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.792528 24726 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.792805 24965 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.793079 24726 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.794049 21467 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.794790 25035 ts_tablet_manager.cc:927] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.794934 25035 tablet_copy_client.cc:323] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.794862 21189 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.795161 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 122762e845a046f4a84afeeacc01b42f from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:15.795202 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 122762e845a046f4a84afeeacc01b42f from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-122762e845a046f4a84afeeacc01b42f
I20250110 03:07:15.795667 21488 tablet_copy_source_session.cc:215] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.795811 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:15.795854 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.796368 21730 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.796758 24991 tablet_copy_client.cc:538] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.796785 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 122762e845a046f4a84afeeacc01b42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.797382 24991 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.797757 24769 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.797956 25035 tablet_copy_client.cc:806] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.798063 25035 tablet_copy_client.cc:670] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.798069 23656 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:07:15.799050 21273 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62aacc8deebd43debd76a2c10815da2f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.799264 21730 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.799885 24976 ts_tablet_manager.cc:927] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.800050 24976 tablet_copy_client.cc:323] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.797961 24769 tablet_bootstrap.cc:492] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.800226 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57273)
I20250110 03:07:15.800282 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57273: session id = d0be0af85dee41dd8d01ef4b2af7726f-41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:15.800302 24769 ts_tablet_manager.cc:1397] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20250110 03:07:15.800446 24769 raft_consensus.cc:357] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.800505 24769 raft_consensus.cc:738] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.800529 24769 consensus_queue.cc:260] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.800626 24769 ts_tablet_manager.cc:1428] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.800796 21620 tablet_copy_source_session.cc:215] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.800889 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-a6000ace3863451db3fdd8d150cd0944 received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:15.800931 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-a6000ace3863451db3fdd8d150cd0944 on tablet a6000ace3863451db3fdd8d150cd0944 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.801297 25035 tablet_copy_client.cc:538] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.801554 24976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.801546 25120 tablet_replica.cc:331] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:15.801723 25120 raft_consensus.cc:2238] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:15.801816 25120 raft_consensus.cc:2267] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:15.801913 25035 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.802449 25120 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:15.802846 24976 tablet_copy_client.cc:806] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.802951 24976 tablet_copy_client.cc:670] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.803350 24152 raft_consensus.cc:1059] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:15.803499 24152 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.803756 24152 raft_consensus.cc:1059] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:15.803875 24152 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.804306 25120 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:15.804358 25120 log.cc:1198] T b5064ab85103488f873fffef55fd78fd P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/b5064ab85103488f873fffef55fd78fd
I20250110 03:07:15.804703 25120 tablet_replica.cc:331] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:15.804755 25120 raft_consensus.cc:2238] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:15.804826 25120 raft_consensus.cc:2267] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:15.805047 21467 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.805078 21730 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:15.805071 21273 consensus_peers.cc:487] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a1a954889a9452f8a8ee55aa0a6f26e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.805132 21469 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.805279 24726 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.805346 24168 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.805437 24726 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.805675 21597 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.805786 24965 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.805995 24965 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.805909 22969 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.806085 24965 ts_tablet_manager.cc:1397] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250110 03:07:15.806201 24965 raft_consensus.cc:357] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.806273 24965 raft_consensus.cc:738] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.806265 25120 ts_tablet_manager.cc:1905] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:15.806303 24965 consensus_queue.cc:260] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.806838 22970 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.806998 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet b5064ab85103488f873fffef55fd78fd (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:15.807466 25120 ts_tablet_manager.cc:1918] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:15.807513 25120 log.cc:1198] T 250f547c54a84a35b2850d50b897810e P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/250f547c54a84a35b2850d50b897810e
I20250110 03:07:15.807153 24965 ts_tablet_manager.cc:1428] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:15.807659 22971 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.807804 22968 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.807871 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 250f547c54a84a35b2850d50b897810e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
W20250110 03:07:15.808063 21273 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62aacc8deebd43debd76a2c10815da2f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.808203 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-122762e845a046f4a84afeeacc01b42f received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:15.808248 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-122762e845a046f4a84afeeacc01b42f on tablet 122762e845a046f4a84afeeacc01b42f with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.808542 24726 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.809597 24976 tablet_copy_client.cc:538] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.809729 21467 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.810276 24976 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.811030 24168 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.811141 24168 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.811190 24168 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.811318 21730 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.812695 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.812942 24991 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.812943 21193 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.815073 25035 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.815174 24964 ts_tablet_manager.cc:927] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.815306 24964 tablet_copy_client.cc:323] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.815546 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet c5d0922741634918830ee2c163cbce24 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57273)
I20250110 03:07:15.815615 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet c5d0922741634918830ee2c163cbce24 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57273: session id = d0be0af85dee41dd8d01ef4b2af7726f-c5d0922741634918830ee2c163cbce24
I20250110 03:07:15.815915 22968 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.816152 21620 tablet_copy_source_session.cc:215] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.816586 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.817013 25117 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.817701 24964 tablet_copy_client.cc:806] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.817812 24964 tablet_copy_client.cc:670] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.818109 21194 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.818210 21597 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.818670 21467 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.820569 21338 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
W20250110 03:07:15.821134 21273 consensus_peers.cc:487] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a1a954889a9452f8a8ee55aa0a6f26e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.821532 22968 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.821843 21597 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.822032 25177 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.822209 24152 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.822846 24152 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.823833 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5a1a954889a9452f8a8ee55aa0a6f26e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:15.823925 21187 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:15.823988 21597 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.824792 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1a954889a9452f8a8ee55aa0a6f26e with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:15.825716 22968 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.826378 25197 tablet_replica.cc:331] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:15.827117 25197 raft_consensus.cc:2238] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:15.828053 25197 raft_consensus.cc:2267] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:15.828545 25197 ts_tablet_manager.cc:1905] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:15.829552 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.830282 21730 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:15.830343 21404 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:15.830459 21402 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.830703 24991 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.830699 25088 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.830806 24991 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250110 03:07:15.830931 24991 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.831017 24991 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.831049 24991 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.831149 21338 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:15.831168 24991 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.831363 25035 tablet_bootstrap.cc:492] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.831408 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-58e33e63f48f4b758c19774888345ee0 received from {username='slave'} at 127.14.19.135:36717
I20250110 03:07:15.831449 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-58e33e63f48f4b758c19774888345ee0 on tablet 58e33e63f48f4b758c19774888345ee0 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.831470 25035 ts_tablet_manager.cc:1397] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20250110 03:07:15.831591 25035 raft_consensus.cc:357] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.831657 25035 raft_consensus.cc:738] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.831693 25035 consensus_queue.cc:260] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.831872 25035 ts_tablet_manager.cc:1428] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.832051 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-122762e845a046f4a84afeeacc01b42f received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:15.832103 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-122762e845a046f4a84afeeacc01b42f on tablet 122762e845a046f4a84afeeacc01b42f with peer b2e7bce763e24b63bc760b0b82b53454
W20250110 03:07:15.832110 21403 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.832226 24065 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:15.832360 24065 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.833721 21194 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:15.834093 21403 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcde555c3b364cf3b9b582ee3e273f3e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.834304 25014 ts_tablet_manager.cc:927] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.834460 25014 tablet_copy_client.cc:323] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.834684 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:15.834728 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:15.835104 21730 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.835250 21489 tablet_copy_source_session.cc:215] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.836910 25014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.838907 24965 ts_tablet_manager.cc:927] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.839044 24965 tablet_copy_client.cc:323] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.839432 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:15.839484 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:15.839771 21338 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.839941 21489 tablet_copy_source_session.cc:215] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.840497 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.841571 24965 tablet_copy_client.cc:806] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.841666 24965 tablet_copy_client.cc:670] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.842079 25014 tablet_copy_client.cc:806] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.842185 25014 tablet_copy_client.cc:670] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.842489 24965 tablet_copy_client.cc:538] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.843159 24965 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.845479 24976 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.845665 24976 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.845739 24976 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.004s
I20250110 03:07:15.845834 24976 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.845894 24976 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.845921 24976 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.846377 24976 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:15.846927 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.846978 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-41fa30056c0a4b30ab8fcaa14645e7a4 received from {username='slave'} at 127.14.19.136:57273
I20250110 03:07:15.847035 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-41fa30056c0a4b30ab8fcaa14645e7a4 on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.849473 24965 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.849581 25197 ts_tablet_manager.cc:1918] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:15.849642 25197 log.cc:1198] T 5a1a954889a9452f8a8ee55aa0a6f26e P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:15.849675 24965 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.849756 24965 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20250110 03:07:15.849867 24965 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.849932 24965 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.849959 24965 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.850004 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 5a1a954889a9452f8a8ee55aa0a6f26e (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:15.850257 24965 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:15.850452 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-0760e9b8e3c642f09c3287ca586a2a1d received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:15.850498 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-0760e9b8e3c642f09c3287ca586a2a1d on tablet 0760e9b8e3c642f09c3287ca586a2a1d with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.851888 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.852885 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.856211 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.856937 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.857421 25014 tablet_copy_client.cc:538] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.858270 25014 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.858906 24964 tablet_copy_client.cc:538] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.859688 24964 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:15.860605 23101 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.867844 24964 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.868041 24964 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.868139 24964 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250110 03:07:15.868268 24964 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.868417 24964 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.868458 24964 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.868693 24964 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:15.868887 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-c5d0922741634918830ee2c163cbce24 received from {username='slave'} at 127.14.19.136:57273
I20250110 03:07:15.868949 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-c5d0922741634918830ee2c163cbce24 on tablet c5d0922741634918830ee2c163cbce24 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.869772 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.871388 25014 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.871927 25014 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.872131 25014 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250110 03:07:15.872792 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.877278 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.878224 23102 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.883041 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.872555 25014 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.883466 25014 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.883562 25014 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.883919 25014 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20250110 03:07:15.885175 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:15.888708 22968 raft_consensus.cc:1212] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:15.889271 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.890452 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.892985 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:15.897707 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.900393 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.891454 23364 raft_consensus.cc:1212] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:15.907545 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-0760e9b8e3c642f09c3287ca586a2a1d received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:15.907603 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-0760e9b8e3c642f09c3287ca586a2a1d on tablet 0760e9b8e3c642f09c3287ca586a2a1d with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.907876 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.912199 24965 ts_tablet_manager.cc:927] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.912334 24965 tablet_copy_client.cc:323] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.912518 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 58e33e63f48f4b758c19774888345ee0 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:15.912581 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 58e33e63f48f4b758c19774888345ee0 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-58e33e63f48f4b758c19774888345ee0
I20250110 03:07:15.913129 21358 tablet_copy_source_session.cc:215] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.913514 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.914952 25035 ts_tablet_manager.cc:927] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.915069 24965 tablet_copy_client.cc:806] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.915105 25035 tablet_copy_client.cc:323] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.915179 24965 tablet_copy_client.cc:670] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.916060 24965 tablet_copy_client.cc:538] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.916694 24965 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.917294 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:15.917363 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:15.917901 21489 tablet_copy_source_session.cc:215] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.918246 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.919806 25035 tablet_copy_client.cc:806] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.919935 25035 tablet_copy_client.cc:670] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.923316 25035 tablet_copy_client.cc:538] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.923939 25035 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:15.927799 25035 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.927995 25035 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:15.928069 25035 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250110 03:07:15.928171 25035 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.928241 25035 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:15.928268 25035 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.928392 25035 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.928719 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-0760e9b8e3c642f09c3287ca586a2a1d received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:15.928776 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-0760e9b8e3c642f09c3287ca586a2a1d on tablet 0760e9b8e3c642f09c3287ca586a2a1d with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:15.931627 24716 ts_tablet_manager.cc:927] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:15.931773 24716 tablet_copy_client.cc:323] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:15.934526 21338 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:15.935436 21597 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250110 03:07:15.935730 21272 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 250f547c54a84a35b2850d50b897810e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.935854 25177 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.936233 23232 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:15.936902 25177 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.937029 25177 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.937156 21597 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.937491 24755 ts_tablet_manager.cc:927] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:15.937616 24755 tablet_copy_client.cc:323] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:15.937496 24965 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.937793 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet ecbd6afd612b4e86b94429633ac15082 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:15.937844 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet ecbd6afd612b4e86b94429633ac15082 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:15.937980 24965 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.938139 24965 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.005s
I20250110 03:07:15.938263 24965 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.938365 24965 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.938397 24965 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.938383 21620 tablet_copy_source_session.cc:215] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.938372 21187 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.938722 24965 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:15.939074 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-58e33e63f48f4b758c19774888345ee0 received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:15.939116 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-58e33e63f48f4b758c19774888345ee0 on tablet 58e33e63f48f4b758c19774888345ee0 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.939160 23232 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:15.939524 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbd6afd612b4e86b94429633ac15082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.939471 21336 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:15.940343 21271 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.940446 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.940737 24755 tablet_copy_client.cc:806] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
W20250110 03:07:15.940739 21272 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.940847 24755 tablet_copy_client.cc:670] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
W20250110 03:07:15.940910 21272 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.941043 25173 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.941733 21467 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.941995 25177 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.942124 25174 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.942361 21467 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.942920 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 58e33e63f48f4b758c19774888345ee0 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:38517)
I20250110 03:07:15.942963 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 58e33e63f48f4b758c19774888345ee0 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:38517: session id = 531cf07bcce6435c9e18185c60b2118d-58e33e63f48f4b758c19774888345ee0
I20250110 03:07:15.942858 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.943363 24755 tablet_copy_client.cc:538] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.943506 21358 tablet_copy_source_session.cc:215] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.943940 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.943989 24755 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:15.945403 24716 tablet_copy_client.cc:806] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.945518 24716 tablet_copy_client.cc:670] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.946297 21187 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:15.947161 24716 tablet_copy_client.cc:538] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.947882 24716 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:15.949779 24755 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.949966 24755 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:15.950042 24755 ts_tablet_manager.cc:1397] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250110 03:07:15.950134 24755 raft_consensus.cc:357] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.950204 24755 raft_consensus.cc:738] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:15.950229 24755 consensus_queue.cc:260] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.950362 25014 ts_tablet_manager.cc:927] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:15.950484 25014 tablet_copy_client.cc:323] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:15.951023 24755 ts_tablet_manager.cc:1428] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 03:07:15.951140 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet abfade13c24b4cebaf1ec204947154c0 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:15.951195 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet abfade13c24b4cebaf1ec204947154c0 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:15.951403 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-ecbd6afd612b4e86b94429633ac15082 received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:15.951463 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-ecbd6afd612b4e86b94429633ac15082 on tablet ecbd6afd612b4e86b94429633ac15082 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:15.951756 21489 tablet_copy_source_session.cc:215] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:15.952492 25014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfade13c24b4cebaf1ec204947154c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:15.953732 25014 tablet_copy_client.cc:806] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:15.953840 25014 tablet_copy_client.cc:670] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:15.954921 21467 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.955634 24716 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.955639 21597 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.955652 21338 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.955825 24716 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
W20250110 03:07:15.955816 21404 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a790eedb619d47c6a3ba7eba194b70c1. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.955919 24716 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20250110 03:07:15.956001 24203 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.956040 24716 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.956132 24716 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:15.956123 24203 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.956169 24716 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.956288 24716 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:15.956506 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-58e33e63f48f4b758c19774888345ee0 received from {username='slave'} at 127.14.19.134:38517
I20250110 03:07:15.956557 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-58e33e63f48f4b758c19774888345ee0 on tablet 58e33e63f48f4b758c19774888345ee0 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:15.956848 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:15.956972 21598 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:15.957401 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250110 03:07:15.957540 21534 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.957602 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.957702 23364 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:15.957932 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:15.958110 24203 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.958245 21338 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.959141 24138 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.959448 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.960294 21187 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.960752 21192 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.960873 23364 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:15.962813 24220 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.963469 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.964588 24220 raft_consensus.cc:1059] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:15.964727 24220 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:15.964941 25014 tablet_copy_client.cc:538] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:15.965286 21338 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.965453 24221 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.965680 25014 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
W20250110 03:07:15.965785 21404 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a790eedb619d47c6a3ba7eba194b70c1. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:15.966145 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.966374 24221 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:15.969130 25089 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.970270 21338 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.970604 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.971846 21192 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.977859 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.981962 25014 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:15.982162 25014 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:15.982260 25014 ts_tablet_manager.cc:1397] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250110 03:07:15.982368 25014 raft_consensus.cc:357] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.982431 25014 raft_consensus.cc:738] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:15.982465 25014 consensus_queue.cc:260] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:15.982863 25014 ts_tablet_manager.cc:1428] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:15.983019 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-abfade13c24b4cebaf1ec204947154c0 received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:15.983062 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-abfade13c24b4cebaf1ec204947154c0 on tablet abfade13c24b4cebaf1ec204947154c0 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:15.984294 21598 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:15.984628 24221 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:15.984880 21598 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.985301 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:15.985760 21598 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:15.988011 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:15.988255 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.993175 23101 raft_consensus.cc:1212] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:15.993999 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:15.995978 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:16.000274 23364 raft_consensus.cc:1212] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.002439 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.006561 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:16.019169 23101 raft_consensus.cc:1212] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.019605 24755 ts_tablet_manager.cc:927] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.019732 24755 tablet_copy_client.cc:323] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.020165 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 87f8afb021a7407089c710f6dd588d03 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:16.020216 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 87f8afb021a7407089c710f6dd588d03 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-87f8afb021a7407089c710f6dd588d03
I20250110 03:07:16.020753 21489 tablet_copy_source_session.cc:215] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.021078 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f8afb021a7407089c710f6dd588d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.022325 24755 tablet_copy_client.cc:806] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.022452 24755 tablet_copy_client.cc:670] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.027983 24755 tablet_copy_client.cc:538] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.028878 24755 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.035302 24755 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.035507 24755 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.035493 23364 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.035629 24755 ts_tablet_manager.cc:1397] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20250110 03:07:16.035748 24755 raft_consensus.cc:357] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.035851 24755 raft_consensus.cc:738] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.035902 24755 consensus_queue.cc:260] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.036180 24755 ts_tablet_manager.cc:1428] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:16.038864 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-87f8afb021a7407089c710f6dd588d03 received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:16.038928 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-87f8afb021a7407089c710f6dd588d03 on tablet 87f8afb021a7407089c710f6dd588d03 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.039278 25035 tablet_replica.cc:331] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.039381 25035 raft_consensus.cc:2238] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.039440 25035 raft_consensus.cc:2267] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.040091 25035 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.041445 25035 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.041494 25035 log.cc:1198] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:16.041718 25035 ts_tablet_manager.cc:927] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.041806 25035 tablet_copy_client.cc:323] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.042806 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet d52b50e3c87b47bead3a25f7255d1c42 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:16.042865 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet d52b50e3c87b47bead3a25f7255d1c42 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:16.043434 21489 tablet_copy_source_session.cc:215] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.043918 25035 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:16.045079 25035 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250110 03:07:16.045174 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.046269 25035 tablet_copy_client.cc:806] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.046481 25035 tablet_copy_client.cc:670] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.047190 25035 tablet_copy_client.cc:538] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.048262 25035 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.048723 24755 ts_tablet_manager.cc:927] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.048848 24755 tablet_copy_client.cc:323] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.049001 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 49c3c5c20dcc4474b207b9c73904219f from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:16.049057 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 49c3c5c20dcc4474b207b9c73904219f from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:16.049520 21489 tablet_copy_source_session.cc:215] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.049912 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.051213 24755 tablet_copy_client.cc:806] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.051337 24755 tablet_copy_client.cc:670] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.053629 25035 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.053831 25035 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.053925 25035 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250110 03:07:16.054042 25035 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.054126 25035 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.054165 25035 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.054270 25035 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.054490 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42 received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:16.054534 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42 on tablet d52b50e3c87b47bead3a25f7255d1c42 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.055703 24755 tablet_copy_client.cc:538] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.056368 24755 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.064186 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.065317 24716 ts_tablet_manager.cc:927] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.068483 24716 tablet_copy_client.cc:323] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.068758 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41db68f089644324af518422d2cad2fc from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:54949)
I20250110 03:07:16.068821 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41db68f089644324af518422d2cad2fc from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:54949: session id = 531cf07bcce6435c9e18185c60b2118d-41db68f089644324af518422d2cad2fc
I20250110 03:07:16.069197 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:16.069324 21620 tablet_copy_source_session.cc:215] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.069578 24755 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.069659 24716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.069802 24755 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.069896 24755 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250110 03:07:16.069993 24755 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.070061 24755 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.070096 24755 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.070209 24755 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.070983 24716 tablet_copy_client.cc:806] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.071110 24716 tablet_copy_client.cc:670] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.073132 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-49c3c5c20dcc4474b207b9c73904219f received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:16.073192 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-49c3c5c20dcc4474b207b9c73904219f on tablet 49c3c5c20dcc4474b207b9c73904219f with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.073438 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:16.075724 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:16.075718 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:16.076889 23364 raft_consensus.cc:1212] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.079284 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.079787 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.087361 24716 tablet_copy_client.cc:538] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.088214 24716 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.090546 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.091997 24716 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.095130 23101 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.104553 24966 ts_tablet_manager.cc:927] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.104684 24966 tablet_copy_client.cc:323] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.104882 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 49c3c5c20dcc4474b207b9c73904219f from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:16.104941 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 49c3c5c20dcc4474b207b9c73904219f from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:16.105584 21489 tablet_copy_source_session.cc:215] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.106169 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.106727 24716 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.106835 24716 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250110 03:07:16.106969 24716 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.107048 24716 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.107084 24716 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.107182 24716 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:16.108282 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-41db68f089644324af518422d2cad2fc received from {username='slave'} at 127.14.19.134:54949
I20250110 03:07:16.108338 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-41db68f089644324af518422d2cad2fc on tablet 41db68f089644324af518422d2cad2fc with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.108922 25035 ts_tablet_manager.cc:927] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.109051 25035 tablet_copy_client.cc:323] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.109230 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41db68f089644324af518422d2cad2fc from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:16.109285 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41db68f089644324af518422d2cad2fc from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-41db68f089644324af518422d2cad2fc
I20250110 03:07:16.109899 21620 tablet_copy_source_session.cc:215] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.110216 24755 ts_tablet_manager.cc:927] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.110339 24755 tablet_copy_client.cc:323] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.110365 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.110715 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet d52b50e3c87b47bead3a25f7255d1c42 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:16.110777 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet d52b50e3c87b47bead3a25f7255d1c42 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:16.111021 24966 tablet_copy_client.cc:806] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.111119 24966 tablet_copy_client.cc:670] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.111348 21489 tablet_copy_source_session.cc:215] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.111665 24755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.111665 25035 tablet_copy_client.cc:806] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.111773 25035 tablet_copy_client.cc:670] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.112165 24966 tablet_copy_client.cc:538] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.112840 24966 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.112999 25035 tablet_copy_client.cc:538] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.113032 24755 tablet_copy_client.cc:806] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.113128 24755 tablet_copy_client.cc:670] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.113660 25035 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.117915 24755 tablet_copy_client.cc:538] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.117931 24966 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.118098 24966 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.118168 24966 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250110 03:07:16.118259 24966 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.118327 24966 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.118353 24966 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.118453 24966 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:16.118479 24755 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.118777 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-49c3c5c20dcc4474b207b9c73904219f received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:16.118821 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-49c3c5c20dcc4474b207b9c73904219f on tablet 49c3c5c20dcc4474b207b9c73904219f with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.119719 24965 ts_tablet_manager.cc:927] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.119836 24965 tablet_copy_client.cc:323] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.120270 25035 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.120299 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41db68f089644324af518422d2cad2fc from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:16.120374 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41db68f089644324af518422d2cad2fc from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-41db68f089644324af518422d2cad2fc
I20250110 03:07:16.120877 25035 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.120880 21620 tablet_copy_source_session.cc:215] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.121016 25035 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250110 03:07:16.121315 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.121476 25035 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.121559 25035 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.121587 25035 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.121814 25035 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.122076 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-41db68f089644324af518422d2cad2fc received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:16.122119 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-41db68f089644324af518422d2cad2fc on tablet 41db68f089644324af518422d2cad2fc with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.122442 24965 tablet_copy_client.cc:806] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.122545 24965 tablet_copy_client.cc:670] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.122869 24755 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.123060 24755 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.123741 24755 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250110 03:07:16.124066 23101 raft_consensus.cc:1212] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.124239 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
W20250110 03:07:16.124859 21404 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.124891 21338 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250110 03:07:16.124925 21402 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.125057 24203 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.124778 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.125175 24755 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.125270 24755 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.125278 22968 raft_consensus.cc:1212] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20250110 03:07:16.125413 21403 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5064ab85103488f873fffef55fd78fd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.125522 24203 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.125329 24755 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.125835 24755 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.126694 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-d52b50e3c87b47bead3a25f7255d1c42 received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:16.126749 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-d52b50e3c87b47bead3a25f7255d1c42 on tablet d52b50e3c87b47bead3a25f7255d1c42 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.126849 24203 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.128332 21196 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.129990 25014 ts_tablet_manager.cc:927] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.130110 25014 tablet_copy_client.cc:323] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.130151 24965 tablet_copy_client.cc:538] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.130302 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet f1e7baec841a4b7e9cb933d786ec2552 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57273)
I20250110 03:07:16.130370 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet f1e7baec841a4b7e9cb933d786ec2552 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57273: session id = d0be0af85dee41dd8d01ef4b2af7726f-f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:16.130769 24965 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.130828 21620 tablet_copy_source_session.cc:215] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.131129 25014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.131924 25014 tablet_copy_client.cc:806] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.132030 25014 tablet_copy_client.cc:670] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.132614 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.133896 25035 ts_tablet_manager.cc:927] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.134006 25035 tablet_copy_client.cc:323] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.134953 21619 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet ecbd6afd612b4e86b94429633ac15082 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:16.134847 21338 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.134999 21619 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet ecbd6afd612b4e86b94429633ac15082 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:16.135506 21619 tablet_copy_source_session.cc:215] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.136029 24991 ts_tablet_manager.cc:927] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.136135 24991 tablet_copy_client.cc:323] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.136207 24965 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.136391 24965 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.136478 24965 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20250110 03:07:16.136580 24965 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.136665 24965 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.136695 24965 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.136785 25014 tablet_copy_client.cc:538] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.136811 24965 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.137154 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-41db68f089644324af518422d2cad2fc received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:16.137228 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-41db68f089644324af518422d2cad2fc on tablet 41db68f089644324af518422d2cad2fc with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.137519 25014 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.137645 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet f1e7baec841a4b7e9cb933d786ec2552 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:16.137687 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet f1e7baec841a4b7e9cb933d786ec2552 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:16.137782 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbd6afd612b4e86b94429633ac15082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.138167 21620 tablet_copy_source_session.cc:215] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.138453 24991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.139752 24991 tablet_copy_client.cc:806] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.139871 24991 tablet_copy_client.cc:670] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.139760 25035 tablet_copy_client.cc:806] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.140071 25035 tablet_copy_client.cc:670] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.141429 25035 tablet_copy_client.cc:538] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.141984 25035 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.143138 25014 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.143474 24966 ts_tablet_manager.cc:927] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.143544 24991 tablet_copy_client.cc:538] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.143601 24966 tablet_copy_client.cc:323] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.143842 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet f1e7baec841a4b7e9cb933d786ec2552 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:54949)
I20250110 03:07:16.143895 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet f1e7baec841a4b7e9cb933d786ec2552 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:54949: session id = 531cf07bcce6435c9e18185c60b2118d-f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:16.144007 25014 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.144129 25014 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250110 03:07:16.144275 25014 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.144341 21620 tablet_copy_source_session.cc:215] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.144438 24991 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.144428 25014 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.144563 25014 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.144621 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.144666 25014 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.144825 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-f1e7baec841a4b7e9cb933d786ec2552 received from {username='slave'} at 127.14.19.136:57273
I20250110 03:07:16.144860 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-f1e7baec841a4b7e9cb933d786ec2552 on tablet f1e7baec841a4b7e9cb933d786ec2552 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.145860 24966 tablet_copy_client.cc:806] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.145969 24966 tablet_copy_client.cc:670] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.146481 24966 tablet_copy_client.cc:538] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.147140 24966 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.149356 21730 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.149554 24991 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.149756 24991 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.149854 24991 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250110 03:07:16.149948 24991 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.150035 24991 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.150063 24991 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.150147 23364 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.150285 24991 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20250110 03:07:16.150686 21338 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.150766 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-f1e7baec841a4b7e9cb933d786ec2552 received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:16.150817 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-f1e7baec841a4b7e9cb933d786ec2552 on tablet f1e7baec841a4b7e9cb933d786ec2552 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.151073 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250110 03:07:16.151216 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.151489 21598 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.151613 24966 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.151787 24966 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.151863 24966 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250110 03:07:16.151973 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.151975 24966 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.152040 24966 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.152065 24966 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
W20250110 03:07:16.152460 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.152513 22579 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.152531 24966 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.153100 22579 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.153211 21598 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.154450 21194 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.154876 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-f1e7baec841a4b7e9cb933d786ec2552 received from {username='slave'} at 127.14.19.134:54949
I20250110 03:07:16.155335 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-f1e7baec841a4b7e9cb933d786ec2552 on tablet f1e7baec841a4b7e9cb933d786ec2552 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.155789 25044 raft_consensus.cc:1059] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.155969 25044 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.156034 25035 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.156214 25035 tablet_bootstrap.cc:492] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.156307 25035 ts_tablet_manager.cc:1397] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250110 03:07:16.156411 25035 raft_consensus.cc:357] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.156492 25035 raft_consensus.cc:738] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.156534 25035 consensus_queue.cc:260] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.156754 21598 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.156819 23364 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.157440 21338 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.157768 23363 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:16.157790 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:16.157873 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.158012 25035 ts_tablet_manager.cc:1428] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:16.158001 24343 consensus_queue.cc:1037] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 1.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.158253 24343 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.158541 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-ecbd6afd612b4e86b94429633ac15082 received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:16.158610 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-ecbd6afd612b4e86b94429633ac15082 on tablet ecbd6afd612b4e86b94429633ac15082 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.158998 25014 ts_tablet_manager.cc:927] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.159118 25014 tablet_copy_client.cc:323] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.159282 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 49c3c5c20dcc4474b207b9c73904219f from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:16.159344 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 49c3c5c20dcc4474b207b9c73904219f from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:16.159830 21489 tablet_copy_source_session.cc:215] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.160146 25014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.159451 25691 consensus_queue.cc:1037] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 1.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.160274 21338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36206
I20250110 03:07:16.160369 21338 raft_consensus.cc:604] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:16.160423 21336 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.161159 25014 tablet_copy_client.cc:806] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.161278 25014 tablet_copy_client.cc:670] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.162019 25014 tablet_copy_client.cc:538] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.162050 24343 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.162710 25014 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.163152 23363 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.163416 21196 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.164670 21336 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.166391 21598 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.167665 21730 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.168428 21598 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:16.168726 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:16.168789 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e11143dad6b402fb0d16c48285646c9. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.168848 24397 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.169055 23363 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.169246 24397 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.169883 24397 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.169999 23363 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.170900 25014 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.170900 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 4e11143dad6b402fb0d16c48285646c9 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.171084 25014 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.171169 25014 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20250110 03:07:16.171293 25014 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.171360 25014 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.171325 21598 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.171388 25014 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.171299 21194 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 7 to 8, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.171782 24964 ts_tablet_manager.cc:927] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:16.171891 24964 tablet_copy_client.cc:323] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:16.172089 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet ea70c2cc77384698aedb06bee2b639ce from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:41729)
I20250110 03:07:16.172142 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet ea70c2cc77384698aedb06bee2b639ce from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:41729: session id = d0be0af85dee41dd8d01ef4b2af7726f-ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:16.172708 21751 tablet_copy_source_session.cc:215] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.172981 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.173115 23363 raft_consensus.cc:1212] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.173141 25014 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:16.173307 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-49c3c5c20dcc4474b207b9c73904219f received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:16.173349 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-49c3c5c20dcc4474b207b9c73904219f on tablet 49c3c5c20dcc4474b207b9c73904219f with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.173520 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11143dad6b402fb0d16c48285646c9 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.173746 25725 tablet_replica.cc:331] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.173810 25725 raft_consensus.cc:2238] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.173883 25725 raft_consensus.cc:2267] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.174084 24964 tablet_copy_client.cc:806] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.174181 24964 tablet_copy_client.cc:670] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.174187 25725 ts_tablet_manager.cc:1905] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.177469 24964 tablet_copy_client.cc:538] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.178169 24964 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.180148 25725 ts_tablet_manager.cc:1918] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:16.180282 25725 log.cc:1198] T 4e11143dad6b402fb0d16c48285646c9 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:16.180694 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 4e11143dad6b402fb0d16c48285646c9 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.182323 24964 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.182534 24964 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.182749 24964 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250110 03:07:16.182839 24964 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.182883 24964 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.182901 24964 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.182976 24964 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.183202 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-ea70c2cc77384698aedb06bee2b639ce received from {username='slave'} at 127.14.19.136:41729
I20250110 03:07:16.183267 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-ea70c2cc77384698aedb06bee2b639ce on tablet ea70c2cc77384698aedb06bee2b639ce with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.184912 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:16.185655 23232 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.185657 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.185838 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.186094 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.186242 23101 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:16.186362 21533 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:16.186609 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.186692 24123 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.186784 24123 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.186998 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.187072 24438 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.188271 21196 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.188472 24123 raft_consensus.cc:1059] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.188473 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.188625 24123 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:16.189724 22971 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.189790 23232 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.189826 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.191030 21730 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.191092 24165 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.191130 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250110 03:07:16.191551 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.191686 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.192190 21729 raft_consensus.cc:867] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
W20250110 03:07:16.192250 21729 raft_consensus.cc:1574] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:16.192369 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
W20250110 03:07:16.192687 21533 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.193882 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.194633 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.194947 24437 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.195436 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.196121 22971 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.196251 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.196396 21189 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.197476 24966 ts_tablet_manager.cc:927] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:16.197592 24966 tablet_copy_client.cc:323] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:16.198887 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:16.199522 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.199931 24138 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.200639 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250110 03:07:16.200788 21533 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.200870 24138 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250110 03:07:16.201056 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.201100 22971 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.201288 24138 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.201870 24138 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.202196 22971 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.202337 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet ea70c2cc77384698aedb06bee2b639ce from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:56637)
I20250110 03:07:16.202381 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet ea70c2cc77384698aedb06bee2b639ce from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:56637: session id = 531cf07bcce6435c9e18185c60b2118d-ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:16.202840 21751 tablet_copy_source_session.cc:215] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.203052 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.203326 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.203564 21189 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.204020 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.204226 24965 ts_tablet_manager.cc:927] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.204186 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.204329 24965 tablet_copy_client.cc:323] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.204641 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 3b266cc7c4f341f785152932febcb79b from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:16.204689 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 3b266cc7c4f341f785152932febcb79b from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-3b266cc7c4f341f785152932febcb79b
I20250110 03:07:16.204696 24966 tablet_copy_client.cc:806] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.204815 24966 tablet_copy_client.cc:670] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.205237 21489 tablet_copy_source_session.cc:215] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.205737 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.205688 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:16.205942 23233 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250110 03:07:16.206256 21533 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.206413 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20250110 03:07:16.206629 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.206681 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.206733 22971 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.206782 24965 tablet_copy_client.cc:806] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.206810 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.206887 24965 tablet_copy_client.cc:670] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.206881 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.207172 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.207993 24438 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.209393 21196 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.209398 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.209654 22971 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.210156 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:16.210350 25129 tablet_replica.cc:331] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:16.210448 25129 raft_consensus.cc:2238] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.210503 25129 pending_rounds.cc:62] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Trying to abort 1 pending ops.
I20250110 03:07:16.210547 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.210588 25129 pending_rounds.cc:69] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7112615674612981760 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "41db68f089644324af518422d2cad2fc" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } } }
I20250110 03:07:16.210793 25129 raft_consensus.cc:2911] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250110 03:07:16.210834 25129 raft_consensus.cc:2883] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:16.210855 25129 raft_consensus.cc:2267] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.210870 24964 ts_tablet_manager.cc:927] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.210973 24964 tablet_copy_client.cc:323] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.210988 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.211351 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57273)
I20250110 03:07:16.211397 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57273: session id = d0be0af85dee41dd8d01ef4b2af7726f-22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:16.211911 21620 tablet_copy_source_session.cc:215] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.212371 23363 raft_consensus.cc:1212] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.213068 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.214318 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:16.214686 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.214877 21318 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:16.214998 22971 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20250110 03:07:16.215184 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.215260 24165 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.215826 24964 tablet_copy_client.cc:806] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.215925 24964 tablet_copy_client.cc:670] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.216054 21467 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.216341 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.216974 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.217166 25753 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.217501 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:16.218061 24438 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.219365 22971 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.219383 21189 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.219710 21469 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.220613 23233 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.220808 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:16.220820 24220 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.221076 21467 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.221290 21336 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:16.221349 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.221485 24220 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.222252 23363 raft_consensus.cc:1212] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.222208 25089 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.222779 23363 raft_consensus.cc:1212] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.223076 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.223179 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3b266cc7c4f341f785152932febcb79b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.223631 23233 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.224021 21190 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.224129 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.224669 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.224700 21469 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.224849 24165 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.224913 24123 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.225201 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.225374 21336 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.225900 24165 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.227005 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.227221 21469 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.229127 23344 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.229787 24965 tablet_copy_client.cc:538] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.230125 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.230325 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:16.230518 24965 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.230743 21190 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250110 03:07:16.231335 21403 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.231671 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35500
W20250110 03:07:16.231942 21181 catalog_manager.cc:4839] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): delete failed for tablet 41db68f089644324af518422d2cad2fc because the tablet was not found. No further retry: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc
I20250110 03:07:16.234748 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:16.235031 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b266cc7c4f341f785152932febcb79b with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:16.235039 24964 tablet_copy_client.cc:538] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.235067 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.235442 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.235623 24438 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.235775 24964 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.235922 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.236626 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:16.238204 25793 tablet_replica.cc:331] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:16.238409 25793 raft_consensus.cc:2238] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.238605 25793 raft_consensus.cc:2267] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.239040 25793 ts_tablet_manager.cc:1905] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.239694 24964 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.239907 24964 tablet_bootstrap.cc:492] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.239996 24964 ts_tablet_manager.cc:1397] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20250110 03:07:16.240108 24964 raft_consensus.cc:357] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.240208 24964 raft_consensus.cc:738] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.240250 24964 consensus_queue.cc:260] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.240469 24964 ts_tablet_manager.cc:1428] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:07:16.240764 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-22fe8e3aabc64a848ffc1e4f21fa1b51 received from {username='slave'} at 127.14.19.136:57273
I20250110 03:07:16.240808 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-22fe8e3aabc64a848ffc1e4f21fa1b51 on tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.242132 25793 ts_tablet_manager.cc:1918] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:16.242199 25793 log.cc:1198] T 3b266cc7c4f341f785152932febcb79b P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/3b266cc7c4f341f785152932febcb79b
I20250110 03:07:16.242936 23233 raft_consensus.cc:1212] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.243810 24966 tablet_copy_client.cc:538] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.244158 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 3b266cc7c4f341f785152932febcb79b (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:16.244508 24966 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.244889 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:16.245754 23233 raft_consensus.cc:1212] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20250110 03:07:16.246637 25751 consensus_queue.cc:1159] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "531cf07bcce6435c9e18185c60b2118d" status { last_received { term: 1 index: 8 } last_committed_idx: 7 last_received_current_leader { term: 1 index: 8 } }
I20250110 03:07:16.247665 25129 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.248617 22971 raft_consensus.cc:1212] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:16.248931 25129 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.248984 25129 log.cc:1198] T 41db68f089644324af518422d2cad2fc P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/41db68f089644324af518422d2cad2fc
I20250110 03:07:16.249655 23102 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.250598 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 41db68f089644324af518422d2cad2fc (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:16.250747 25035 ts_tablet_manager.cc:927] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.250860 25035 tablet_copy_client.cc:323] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.251206 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5a1a954889a9452f8a8ee55aa0a6f26e from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:36717)
I20250110 03:07:16.251272 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5a1a954889a9452f8a8ee55aa0a6f26e from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:36717: session id = b2e7bce763e24b63bc760b0b82b53454-5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:16.251694 21358 tablet_copy_source_session.cc:215] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.251992 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1a954889a9452f8a8ee55aa0a6f26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.253131 25035 tablet_copy_client.cc:806] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.253227 25035 tablet_copy_client.cc:670] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.253866 21336 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.254485 23363 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20250110 03:07:16.254498 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.254776 25176 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250110 03:07:16.254984 21180 catalog_manager.cc:4839] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): delete failed for tablet 41db68f089644324af518422d2cad2fc because the tablet was not found. No further retry: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc
I20250110 03:07:16.255010 24966 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.255195 24966 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.255409 21467 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.255527 22971 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.255589 25176 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.255615 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.255777 25117 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.255877 25117 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.256085 25035 tablet_copy_client.cc:538] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.255292 24966 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20250110 03:07:16.256698 25035 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.256700 24966 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.256794 24966 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.256829 24966 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.256958 24966 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.257339 25176 raft_consensus.cc:1022] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.258481 25175 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.260394 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-ea70c2cc77384698aedb06bee2b639ce received from {username='slave'} at 127.14.19.134:56637
I20250110 03:07:16.260334 23363 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.260445 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-ea70c2cc77384698aedb06bee2b639ce on tablet ea70c2cc77384698aedb06bee2b639ce with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.260398 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.261026 24965 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.261224 24965 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.261477 25174 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.261543 21467 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.261623 25174 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.261587 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.262322 24965 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
W20250110 03:07:16.262732 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.262975 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.263103 24965 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.263235 21467 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.263258 25177 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.263311 24965 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.263401 25177 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.263536 23363 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.263451 24965 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.263779 25177 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.263886 24965 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:16.264230 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-3b266cc7c4f341f785152932febcb79b received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:16.264279 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-3b266cc7c4f341f785152932febcb79b on tablet 3b266cc7c4f341f785152932febcb79b with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.264402 25035 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.264576 25035 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.264669 25035 ts_tablet_manager.cc:1397] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20250110 03:07:16.264789 25035 raft_consensus.cc:357] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.264871 25035 raft_consensus.cc:738] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.264907 25035 consensus_queue.cc:260] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.265082 25796 tablet_replica.cc:331] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.265170 25035 ts_tablet_manager.cc:1428] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:16.265493 25796 raft_consensus.cc:2238] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.265571 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-5a1a954889a9452f8a8ee55aa0a6f26e received from {username='slave'} at 127.14.19.135:36717
I20250110 03:07:16.265627 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-5a1a954889a9452f8a8ee55aa0a6f26e on tablet 5a1a954889a9452f8a8ee55aa0a6f26e with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.265615 25796 raft_consensus.cc:2267] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.266054 21336 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36282
I20250110 03:07:16.266121 21336 raft_consensus.cc:604] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:16.266175 25177 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.266419 25796 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.267020 22968 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.267364 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.267792 25796 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.267786 24168 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.267925 25796 log.cc:1198] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/41db68f089644324af518422d2cad2fc
I20250110 03:07:16.268014 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.268117 22968 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.269014 22968 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.269264 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 41db68f089644324af518422d2cad2fc (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:16.269940 25174 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.269940 23363 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.270097 25174 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.270658 25035 tablet_replica.cc:331] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.270720 25035 raft_consensus.cc:2238] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.270799 25035 raft_consensus.cc:2267] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.270956 25035 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.271138 23363 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:16.271149 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.271281 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.271404 22968 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.271478 24168 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.271693 22504 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.271782 22504 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.272267 25035 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.272334 25035 log.cc:1198] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:16.272702 25035 ts_tablet_manager.cc:927] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:16.272676 22504 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.272836 25035 tablet_copy_client.cc:323] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:16.273042 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet ea70c2cc77384698aedb06bee2b639ce from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:38895)
I20250110 03:07:16.273090 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet ea70c2cc77384698aedb06bee2b639ce from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:38895: session id = b2e7bce763e24b63bc760b0b82b53454-ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:16.273626 21751 tablet_copy_source_session.cc:215] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.273696 22968 raft_consensus.cc:1212] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.273965 25035 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:16.274011 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.274335 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.274633 21469 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.274941 25175 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.275161 21336 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.275213 23363 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.275281 21469 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.275585 22968 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.275902 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.276062 25177 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.276105 21469 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
W20250110 03:07:16.276224 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.276337 22971 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.276475 25035 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250110 03:07:16.276552 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.276765 25174 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.276958 23364 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.277369 25035 tablet_copy_client.cc:806] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.277359 25177 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.277477 25035 tablet_copy_client.cc:670] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.277640 24168 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.277998 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.278450 22968 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.278427 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.278923 25035 tablet_copy_client.cc:538] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.278934 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:16.279603 25035 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.279700 21190 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.280619 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.280735 25197 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.280800 25197 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.280869 25197 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.281220 25197 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.282428 25197 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.282477 25197 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:16.284155 25035 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.284351 25035 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.284436 25035 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250110 03:07:16.284543 25035 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.284642 25035 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.284675 25035 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.284790 25035 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.284904 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:16.286360 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-ea70c2cc77384698aedb06bee2b639ce received from {username='slave'} at 127.14.19.135:38895
I20250110 03:07:16.286427 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-ea70c2cc77384698aedb06bee2b639ce on tablet ea70c2cc77384698aedb06bee2b639ce with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.288658 21336 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.289213 23364 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.289441 25177 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250110 03:07:16.289995 21273 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.290374 22968 raft_consensus.cc:1212] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.290375 22971 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.290712 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250110 03:07:16.290931 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.290992 24152 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.291083 24152 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.291688 25177 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.292651 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.292883 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.293803 21190 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.293854 22968 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.296347 21336 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.296893 21598 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.297065 23364 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.297271 25174 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20250110 03:07:16.297355 21272 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.297442 25176 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20250110 03:07:16.297554 21273 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.297812 25174 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.298151 21598 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.298755 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.299090 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.299314 21190 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.300038 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:16.301757 25867 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:16.301877 25867 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.301954 25867 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.302227 25867 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.304813 25867 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:16.304939 25867 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:16.305176 25176 raft_consensus.cc:988] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: : Instructing follower d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:16.305227 25176 raft_consensus.cc:1076] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Signalling peer d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:16.305487 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:16.305517 23364 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
from {username='slave'} at 127.14.19.129:39847
I20250110 03:07:16.305603 23364 raft_consensus.cc:491] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:16.305656 23364 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.306185 23364 raft_consensus.cc:513] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.306438 23364 leader_election.cc:290] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.306681 21598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:16.306740 21598 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.307279 21598 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:16.308066 21729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:16.308137 21729 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.308864 21729 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:16.309391 23300 leader_election.cc:304] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb, d0be0af85dee41dd8d01ef4b2af7726f; no voters:
I20250110 03:07:16.309499 24665 raft_consensus.cc:2798] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:16.309514 23233 raft_consensus.cc:1212] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.309589 24665 raft_consensus.cc:695] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Becoming Leader. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Running, Role: LEADER
I20250110 03:07:16.309612 25035 ts_tablet_manager.cc:927] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.309667 24665 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.309739 25035 tablet_copy_client.cc:323] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.309732 22884 raft_consensus.cc:1059] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.310314 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet fcde555c3b364cf3b9b582ee3e273f3e from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:16.310360 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet fcde555c3b364cf3b9b582ee3e273f3e from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:16.310400 22884 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.310968 21489 tablet_copy_source_session.cc:215] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.311112 22968 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.311182 21336 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.311285 21597 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.311247 21196 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: term changed from 1 to 2, leader changed from a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) to d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136). New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.311340 23102 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.311553 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.311667 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.311730 22884 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.311867 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.312678 21336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44bb3816ff5a4d44a249b8bbabdc7b04" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:16.312743 21336 raft_consensus.cc:3049] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:16.312768 21336 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:16.312824 22884 raft_consensus.cc:1022] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer e4d83809ce4746d5ad65e8462857fd9c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.312810 21336 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.312899 21336 raft_consensus.cc:3054] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.313186 25035 tablet_copy_client.cc:806] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.313289 25035 tablet_copy_client.cc:670] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.313625 21336 raft_consensus.cc:2463] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:16.313913 22874 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.314649 22874 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.314960 21597 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.315042 21598 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.315613 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.315716 21469 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.315814 22561 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.315922 23102 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.315955 22968 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.316129 21336 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.316792 21196 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.316967 24437 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.317117 23233 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.317512 25035 tablet_copy_client.cc:538] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.317802 21729 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.317936 24437 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.317992 25753 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.318149 25035 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.318548 25753 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.318876 23364 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.318946 21729 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.319491 25753 raft_consensus.cc:1059] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.319644 25753 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.320037 21195 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.320835 21469 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:16.321121 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.321290 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.322078 21467 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.322337 23233 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.322511 23102 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.322608 23100 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.322916 23100 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.322926 21338 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.323211 23364 raft_consensus.cc:1212] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.323611 24437 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.323714 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:16.323994 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.324090 24437 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.324158 24203 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.324275 24203 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.324594 21598 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.324570 23100 raft_consensus.cc:867] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.324831 25089 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:16.324820 23100 raft_consensus.cc:1574] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.325011 23100 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.325147 23233 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.325510 21598 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.325965 21467 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.326048 21729 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.326328 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.326954 23102 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.327030 23232 raft_consensus.cc:1212] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.327389 24437 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.327641 21336 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.327728 21729 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.327950 21189 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.328184 22874 raft_consensus.cc:1022] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer e4d83809ce4746d5ad65e8462857fd9c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.328251 22874 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.328678 21469 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.328781 22971 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.328963 25176 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.329061 23364 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.329448 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 87f8afb021a7407089c710f6dd588d03 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.329520 25176 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.329883 24438 raft_consensus.cc:1059] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.330022 24438 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.330163 21194 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.330680 23233 raft_consensus.cc:1212] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.330833 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 62aacc8deebd43debd76a2c10815da2f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.330963 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f8afb021a7407089c710f6dd588d03 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:16.331032 25867 tablet_replica.cc:331] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:16.331073 25867 raft_consensus.cc:2238] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.331089 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.331086 23232 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.331107 25867 raft_consensus.cc:2267] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.331241 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.331275 21469 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.331454 25867 ts_tablet_manager.cc:1905] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.331528 23233 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.331609 24726 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.331947 21467 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.332082 21194 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.332410 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.332806 21468 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:16.332864 21469 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.332930 21336 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.332967 22971 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.333078 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250110 03:07:16.334055 21273 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62aacc8deebd43debd76a2c10815da2f. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:16.334511 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.334753 23101 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.334769 21336 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.334981 21729 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.334960 24221 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.335081 24221 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.335172 24123 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.335278 25867 ts_tablet_manager.cc:1918] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.335345 25867 log.cc:1198] T 87f8afb021a7407089c710f6dd588d03 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/87f8afb021a7407089c710f6dd588d03
I20250110 03:07:16.335721 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 87f8afb021a7407089c710f6dd588d03 (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:16.336476 24123 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.336620 22884 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.337612 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 62aacc8deebd43debd76a2c10815da2f with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.337545 21729 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.337836 23233 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.338158 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 87f8afb021a7407089c710f6dd588d03 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.338390 23363 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.338907 23101 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.339071 21336 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.339931 25035 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.340116 25035 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.340209 25035 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250110 03:07:16.340332 21194 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.340669 25921 tablet_replica.cc:331] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.340773 25921 raft_consensus.cc:2238] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.340731 21187 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.340914 25921 raft_consensus.cc:2267] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.340970 23364 raft_consensus.cc:1212] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:16.340816 25035 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.341066 25035 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.341131 25035 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.341292 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f8afb021a7407089c710f6dd588d03 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:16.341470 25793 tablet_replica.cc:331] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:16.341470 25035 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.341532 25793 raft_consensus.cc:2238] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.341612 25793 raft_consensus.cc:2267] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.341872 23102 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.341979 25793 ts_tablet_manager.cc:1905] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.342082 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.342626 23102 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.342658 21729 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.342831 25753 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250110 03:07:16.342907 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.342960 24438 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.343380 25793 ts_tablet_manager.cc:1918] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:16.343433 25793 log.cc:1198] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/87f8afb021a7407089c710f6dd588d03
I20250110 03:07:16.343552 23102 raft_consensus.cc:867] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:16.343604 23102 raft_consensus.cc:1574] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.343691 23102 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.343792 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-fcde555c3b364cf3b9b582ee3e273f3e received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:16.343806 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 87f8afb021a7407089c710f6dd588d03 (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:16.343830 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-fcde555c3b364cf3b9b582ee3e273f3e on tablet fcde555c3b364cf3b9b582ee3e273f3e with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.344434 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.344525 23233 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.344527 21468 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.345183 24438 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.345810 25753 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.345953 25921 ts_tablet_manager.cc:1905] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.345999 24438 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.346561 23364 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.347337 21729 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.347805 25921 ts_tablet_manager.cc:1918] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.347828 24964 ts_tablet_manager.cc:927] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.347909 25921 log.cc:1198] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:16.347934 24964 tablet_copy_client.cc:323] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.348151 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet fcde555c3b364cf3b9b582ee3e273f3e from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:16.348199 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet fcde555c3b364cf3b9b582ee3e273f3e from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:16.348232 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f1e7baec841a4b7e9cb933d786ec2552 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.348301 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 62aacc8deebd43debd76a2c10815da2f (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:16.348675 21489 tablet_copy_source_session.cc:215] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.348877 21195 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.349030 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.349349 23102 raft_consensus.cc:867] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:16.349396 23102 raft_consensus.cc:1574] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.349498 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.349474 23102 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.349581 25197 tablet_replica.cc:331] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.349632 25197 raft_consensus.cc:2238] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.349709 25197 raft_consensus.cc:2267] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.349980 25197 ts_tablet_manager.cc:1905] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.350212 24964 tablet_copy_client.cc:806] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.350307 24964 tablet_copy_client.cc:670] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.350941 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.351311 25197 ts_tablet_manager.cc:1918] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:16.351356 25197 log.cc:1198] T f1e7baec841a4b7e9cb933d786ec2552 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/f1e7baec841a4b7e9cb933d786ec2552
W20250110 03:07:16.351768 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.352133 23233 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.352386 23100 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.352713 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet f1e7baec841a4b7e9cb933d786ec2552 (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:16.353024 23232 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.353195 25753 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.353313 25754 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.353780 23100 raft_consensus.cc:867] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:16.353818 23100 raft_consensus.cc:1574] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.353884 23100 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.353821 25754 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.354689 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.354862 24964 tablet_copy_client.cc:538] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.355098 25754 raft_consensus.cc:1059] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.355226 25754 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.355626 24964 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.355849 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f1e7baec841a4b7e9cb933d786ec2552 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.356032 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.356294 21190 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted, now has a pending config: VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131), VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134), VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135), VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136), NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.356514 25753 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.357035 23364 raft_consensus.cc:867] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:16.357097 23364 raft_consensus.cc:1574] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 12' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.357204 23364 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.357968 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35032
W20250110 03:07:16.358361 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.358455 25921 tablet_replica.cc:331] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.358592 25921 raft_consensus.cc:2238] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.358489 23100 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.358760 25921 raft_consensus.cc:2267] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.359190 23232 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.360350 23233 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.360724 22561 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.361348 23102 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.361732 25921 ts_tablet_manager.cc:1905] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.362962 25921 ts_tablet_manager.cc:1918] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:16.363029 25921 log.cc:1198] T f1e7baec841a4b7e9cb933d786ec2552 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:16.363090 22561 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.363286 22561 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.363370 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet f1e7baec841a4b7e9cb933d786ec2552 (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:16.363579 23364 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.364436 23102 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.364606 23233 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.364952 24964 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.365130 24964 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.365217 24964 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250110 03:07:16.365386 24964 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.365355 21195 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131), VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134), VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135), VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136), NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.365471 24964 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.365504 24964 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.365671 24964 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.365832 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-fcde555c3b364cf3b9b582ee3e273f3e received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:16.365878 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-fcde555c3b364cf3b9b582ee3e273f3e on tablet fcde555c3b364cf3b9b582ee3e273f3e with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.366241 24965 ts_tablet_manager.cc:927] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.366371 24965 tablet_copy_client.cc:323] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.366720 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet fcde555c3b364cf3b9b582ee3e273f3e from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:16.366775 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet fcde555c3b364cf3b9b582ee3e273f3e from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:16.367290 21489 tablet_copy_source_session.cc:215] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.367486 21468 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.367508 21597 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.367904 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:16.368114 21404 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a790eedb619d47c6a3ba7eba194b70c1. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.368161 23233 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.368371 24138 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.368611 24138 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.368678 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.368701 21597 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.368865 24220 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.368988 24221 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.369264 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.369513 23102 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.369568 24138 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.369752 24122 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.370064 23102 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.370610 24965 tablet_copy_client.cc:806] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.370718 24965 tablet_copy_client.cc:670] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.370788 23233 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.370824 24221 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.371929 21190 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 12 to 13, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.372150 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a790eedb619d47c6a3ba7eba194b70c1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.372258 23363 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.372761 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49544
W20250110 03:07:16.373066 21181 catalog_manager.cc:4839] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): delete failed for tablet f1e7baec841a4b7e9cb933d786ec2552 because the tablet was not found. No further retry: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:16.373095 24965 tablet_copy_client.cc:538] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.373172 21189 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.373740 24965 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.374343 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f1e7baec841a4b7e9cb933d786ec2552 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.374356 21598 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.374493 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet a790eedb619d47c6a3ba7eba194b70c1 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.374660 25725 tablet_replica.cc:331] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.374642 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.374722 25725 raft_consensus.cc:2238] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.374788 25725 raft_consensus.cc:2267] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.375149 25725 ts_tablet_manager.cc:1905] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.376533 25725 ts_tablet_manager.cc:1918] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.376603 25725 log.cc:1198] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:16.377386 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet a790eedb619d47c6a3ba7eba194b70c1 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:16.379078 24965 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.379279 24965 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.379380 24965 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250110 03:07:16.379508 24965 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.379618 24965 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.379662 24965 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.379789 24965 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.380048 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-fcde555c3b364cf3b9b582ee3e273f3e received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:16.380101 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-fcde555c3b364cf3b9b582ee3e273f3e on tablet fcde555c3b364cf3b9b582ee3e273f3e with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.387740 23102 raft_consensus.cc:1212] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.390264 26007 ts_tablet_manager.cc:927] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.390415 26007 tablet_copy_client.cc:323] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.390744 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer c57c9dfa476f4d21b79f81114c5d74cb ({username='slave'} at 127.14.19.131:41927)
I20250110 03:07:16.390805 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer c57c9dfa476f4d21b79f81114c5d74cb at {username='slave'} at 127.14.19.131:41927: session id = c57c9dfa476f4d21b79f81114c5d74cb-7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:16.391358 21358 tablet_copy_source_session.cc:215] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.393463 26007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.396051 26007 tablet_copy_client.cc:806] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.396189 26007 tablet_copy_client.cc:670] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.397047 22971 raft_consensus.cc:1212] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.396786 26007 tablet_copy_client.cc:538] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.398484 26007 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:16.403863 26007 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.404994 26007 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap complete.
I20250110 03:07:16.405159 26007 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250110 03:07:16.405359 26007 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.405848 26007 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: LEARNER
I20250110 03:07:16.406051 26007 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.406450 26007 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.406692 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-7dbef1612c8b4e2baa670b1cbe0e8960 received from {username='slave'} at 127.14.19.131:41927
I20250110 03:07:16.406735 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-7dbef1612c8b4e2baa670b1cbe0e8960 on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with peer c57c9dfa476f4d21b79f81114c5d74cb
I20250110 03:07:16.413481 24966 ts_tablet_manager.cc:927] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.413616 24966 tablet_copy_client.cc:323] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.413877 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:38517)
I20250110 03:07:16.413929 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:38517: session id = 531cf07bcce6435c9e18185c60b2118d-7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:16.414469 21358 tablet_copy_source_session.cc:215] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.414956 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.416268 24966 tablet_copy_client.cc:806] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.416376 24966 tablet_copy_client.cc:670] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.417351 24966 tablet_copy_client.cc:538] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.417977 24966 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.422093 24966 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.422291 24966 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.422391 24966 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250110 03:07:16.423617 22971 raft_consensus.cc:1212] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.422535 24966 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.424033 24966 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.424115 24966 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.424443 24966 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:16.424610 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-7dbef1612c8b4e2baa670b1cbe0e8960 received from {username='slave'} at 127.14.19.134:38517
I20250110 03:07:16.424667 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-7dbef1612c8b4e2baa670b1cbe0e8960 on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.431461 21336 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36388
I20250110 03:07:16.431538 21336 raft_consensus.cc:604] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:16.433063 24220 raft_consensus.cc:1059] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.433234 24220 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.433771 23102 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.433892 23364 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.434033 24203 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.434088 21597 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.434290 24203 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.434631 21730 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.435040 24203 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.435150 24203 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.435899 24203 raft_consensus.cc:1022] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.436831 24220 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.436992 23364 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.437557 21729 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.438189 21189 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.438261 21597 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.438779 23102 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.439721 21468 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.440419 23364 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.440711 23233 raft_consensus.cc:1212] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.441006 21597 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.441121 24065 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.441342 24065 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.441562 21729 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.441715 24065 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.442658 24065 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.443572 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 80152844286545c68c7656ef9bbe8604 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.443670 23364 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.443985 21597 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.444296 21729 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.444908 21195 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 6 to 7, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.445358 24966 ts_tablet_manager.cc:927] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.445492 24966 tablet_copy_client.cc:323] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.445672 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet fcde555c3b364cf3b9b582ee3e273f3e from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:16.445722 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet fcde555c3b364cf3b9b582ee3e273f3e from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:16.446297 21489 tablet_copy_source_session.cc:215] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.446579 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 80152844286545c68c7656ef9bbe8604 with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:16.446660 25129 tablet_replica.cc:331] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:16.446710 25129 raft_consensus.cc:2238] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.446729 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.446749 25129 raft_consensus.cc:2267] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.447198 25129 ts_tablet_manager.cc:1905] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.448576 24966 tablet_copy_client.cc:806] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.448845 24966 tablet_copy_client.cc:670] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.450583 25129 ts_tablet_manager.cc:1918] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.450634 25129 log.cc:1198] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/80152844286545c68c7656ef9bbe8604
I20250110 03:07:16.451057 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 80152844286545c68c7656ef9bbe8604 (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:16.454607 24966 tablet_copy_client.cc:538] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.455246 24966 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.456867 23364 raft_consensus.cc:1212] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.460834 24966 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.461403 24966 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.461567 24966 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20250110 03:07:16.461764 21597 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.461825 24966 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.462245 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.462339 23364 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.462528 24123 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.462677 24123 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250110 03:07:16.462761 21534 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:16.463145 21534 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81b3bd116ad54a9fa10d2d0c829bb467. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.463354 25753 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.464250 23364 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.464962 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.465359 21189 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.466434 23233 raft_consensus.cc:1212] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.468158 24966 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.468253 24966 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.468482 24966 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.007s user 0.000s sys 0.000s
I20250110 03:07:16.469178 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-fcde555c3b364cf3b9b582ee3e273f3e received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:16.469228 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-fcde555c3b364cf3b9b582ee3e273f3e on tablet fcde555c3b364cf3b9b582ee3e273f3e with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.485311 24220 raft_consensus.cc:1059] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.485478 24220 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.485944 21729 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.486109 22884 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.486215 23364 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.486385 21597 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.486608 23102 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.486874 22884 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.486961 22874 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.487046 22884 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.488433 25089 raft_consensus.cc:1022] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.488651 22884 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.490021 23364 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.490381 23102 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.491225 24065 raft_consensus.cc:1059] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.491215 21196 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.491223 21729 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.491348 24065 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.492007 23364 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.492179 25089 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.492441 21730 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.492709 23102 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.492771 22874 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.493119 25089 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:16.493546 21182 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet abfade13c24b4cebaf1ec204947154c0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.493983 25089 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.494777 23102 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.495370 23364 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.495396 21190 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.496255 21730 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.496569 21468 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.496976 23364 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.497220 21729 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.497431 23102 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.497592 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.497704 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.497756 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.498363 24220 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.498562 23364 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.499667 23102 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.499778 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet abfade13c24b4cebaf1ec204947154c0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.500025 21729 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.501132 21195 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.502007 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet abfade13c24b4cebaf1ec204947154c0 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:16.502295 21468 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.502494 25793 tablet_replica.cc:331] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:16.502614 25793 raft_consensus.cc:2238] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.502671 25793 pending_rounds.cc:62] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Trying to abort 1 pending ops.
I20250110 03:07:16.502722 25793 pending_rounds.cc:69] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7112615675844190208 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "abfade13c24b4cebaf1ec204947154c0" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } } }
I20250110 03:07:16.502831 25793 raft_consensus.cc:2911] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20250110 03:07:16.502866 25793 raft_consensus.cc:2883] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:16.502894 25793 raft_consensus.cc:2267] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.503072 23102 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.503340 25089 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250110 03:07:16.503412 26275 consensus_queue.cc:1159] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20250110 03:07:16.503731 25793 ts_tablet_manager.cc:1905] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.504469 23364 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.504674 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.505090 22971 raft_consensus.cc:1212] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.505211 25793 ts_tablet_manager.cc:1918] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.505252 25793 log.cc:1198] T abfade13c24b4cebaf1ec204947154c0 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:16.505445 22874 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.505612 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet abfade13c24b4cebaf1ec204947154c0 (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:16.505858 23363 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.506495 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet abfade13c24b4cebaf1ec204947154c0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.506902 24964 ts_tablet_manager.cc:927] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.506855 21190 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.507031 24964 tablet_copy_client.cc:323] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.507297 23100 raft_consensus.cc:1212] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.507339 23102 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.508742 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet abfade13c24b4cebaf1ec204947154c0 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.508843 25921 tablet_replica.cc:331] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.508913 25921 raft_consensus.cc:2238] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.508927 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57367)
I20250110 03:07:16.508970 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57367: session id = d0be0af85dee41dd8d01ef4b2af7726f-7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:16.508980 25921 raft_consensus.cc:2267] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.509560 21358 tablet_copy_source_session.cc:215] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.509896 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.510141 25921 ts_tablet_manager.cc:1905] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.511303 24964 tablet_copy_client.cc:806] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.511440 24964 tablet_copy_client.cc:670] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.511447 25921 ts_tablet_manager.cc:1918] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.511500 25921 log.cc:1198] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:16.512398 24964 tablet_copy_client.cc:538] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.512841 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet abfade13c24b4cebaf1ec204947154c0 (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:16.513165 24964 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.518640 24964 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.518982 24964 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.519147 24964 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20250110 03:07:16.519344 24964 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.519521 24964 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.519608 24964 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.520216 24964 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.520557 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-7dbef1612c8b4e2baa670b1cbe0e8960 received from {username='slave'} at 127.14.19.136:57367
I20250110 03:07:16.520605 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-7dbef1612c8b4e2baa670b1cbe0e8960 on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.537935 24966 ts_tablet_manager.cc:927] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.538070 24966 tablet_copy_client.cc:323] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.538352 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet b5064ab85103488f873fffef55fd78fd from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:16.538404 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet b5064ab85103488f873fffef55fd78fd from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-b5064ab85103488f873fffef55fd78fd
I20250110 03:07:16.539057 21489 tablet_copy_source_session.cc:215] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.539445 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.540822 24966 tablet_copy_client.cc:806] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.540961 24966 tablet_copy_client.cc:670] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.542520 24966 tablet_copy_client.cc:538] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.543288 24966 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.545995 21182 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet abfade13c24b4cebaf1ec204947154c0 with cas_config_opid_index 6: aborting the task: latest config opid_index 9; task opid_index 6
I20250110 03:07:16.547919 24123 raft_consensus.cc:1059] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.548146 24123 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.548861 21468 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.549047 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.549233 23102 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.549320 23233 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.549422 24966 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.549463 23364 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.549607 21336 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.549654 24438 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.549753 24438 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.549861 24966 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.549960 24966 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20250110 03:07:16.550420 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.551637 24965 ts_tablet_manager.cc:927] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.551668 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.551762 24965 tablet_copy_client.cc:323] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.552102 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 250f547c54a84a35b2850d50b897810e from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:16.552155 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 250f547c54a84a35b2850d50b897810e from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-250f547c54a84a35b2850d50b897810e
I20250110 03:07:16.552732 21358 tablet_copy_source_session.cc:215] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.553156 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.553697 24122 raft_consensus.cc:1022] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.554414 24966 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.554497 24965 tablet_copy_client.cc:806] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.554510 24966 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.554617 24965 tablet_copy_client.cc:670] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.554548 24966 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.554701 24966 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250110 03:07:16.554973 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-b5064ab85103488f873fffef55fd78fd received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:16.555014 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-b5064ab85103488f873fffef55fd78fd on tablet b5064ab85103488f873fffef55fd78fd with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.555361 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.556200 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.557790 23344 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:16.563324 21318 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:16.565600 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:16.569440 24965 tablet_copy_client.cc:538] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.570152 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:16.570181 24965 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.570303 24438 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.571383 23102 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.572335 23364 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.572383 21194 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.573854 24438 raft_consensus.cc:1059] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.573987 24438 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.574436 23102 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.574465 21336 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.574716 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.575362 21338 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.575788 23364 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.576038 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.576218 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.576288 24965 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.576499 24965 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.576478 21468 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.576558 23233 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.576598 24965 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20250110 03:07:16.576714 24965 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.576794 24965 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.576833 24965 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.576975 24965 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:16.576987 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.577152 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-250f547c54a84a35b2850d50b897810e received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:16.577211 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-250f547c54a84a35b2850d50b897810e on tablet 250f547c54a84a35b2850d50b897810e with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.577210 23233 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.577481 21467 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.577725 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.578306 25753 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.578490 21467 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.578766 24964 ts_tablet_manager.cc:927] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.578868 24964 tablet_copy_client.cc:323] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.579017 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet b5064ab85103488f873fffef55fd78fd from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:16.579056 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet b5064ab85103488f873fffef55fd78fd from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-b5064ab85103488f873fffef55fd78fd
I20250110 03:07:16.579614 21489 tablet_copy_source_session.cc:215] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.579766 21194 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.580016 24964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.581053 24964 tablet_copy_client.cc:806] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.581151 24964 tablet_copy_client.cc:670] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.581595 23102 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.581822 21447 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20250110 03:07:16.582319 24123 raft_consensus.cc:1059] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.582440 24123 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.583287 23232 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.583346 23363 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.583366 23233 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.583619 24438 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.583715 24438 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.583907 23363 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.586843 23102 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.587091 24438 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.587240 24438 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.587283 21336 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.587396 24438 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.588083 23233 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.588232 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.588411 21336 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.588433 24165 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.588490 23100 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.588532 24165 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.588805 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.588872 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.588973 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.589021 22971 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.589047 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.589205 25754 raft_consensus.cc:1022] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.589237 24138 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.589283 24964 tablet_copy_client.cc:538] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.589217 21336 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.589990 24964 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:16.590598 25753 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.591192 24165 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.591457 24138 raft_consensus.cc:1022] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.592150 23233 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.592223 23363 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.593941 21195 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.594810 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.595438 24964 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.595655 24964 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:16.595775 24964 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250110 03:07:16.595887 22561 raft_consensus.cc:1022] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.595893 24964 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.595980 24964 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:16.596017 24964 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.596114 21336 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.596477 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.597056 23233 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.597256 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.597295 21468 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.597544 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.598031 21467 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.599220 22951 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:16.600816 23363 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.601006 25754 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.601680 21467 raft_consensus.cc:867] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
W20250110 03:07:16.601716 21467 raft_consensus.cc:1574] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.601724 25754 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.601801 21467 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.602489 24964 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250110 03:07:16.602595 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.602663 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-b5064ab85103488f873fffef55fd78fd received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:16.602699 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-b5064ab85103488f873fffef55fd78fd on tablet b5064ab85103488f873fffef55fd78fd with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:16.602756 23100 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.602941 24165 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.602983 23233 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.603298 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.603324 21467 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.604115 21194 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 9 to 10, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.604746 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.604871 25725 tablet_replica.cc:331] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.604930 25725 raft_consensus.cc:2238] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.605001 25725 raft_consensus.cc:2267] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.605029 24966 ts_tablet_manager.cc:927] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:16.605163 24966 tablet_copy_client.cc:323] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:16.605305 25725 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.605348 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 4e11143dad6b402fb0d16c48285646c9 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:56637)
I20250110 03:07:16.605396 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 4e11143dad6b402fb0d16c48285646c9 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:56637: session id = 531cf07bcce6435c9e18185c60b2118d-4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:16.605615 23363 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.605917 21751 tablet_copy_source_session.cc:215] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.606281 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e11143dad6b402fb0d16c48285646c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.606695 25725 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:16.606743 25725 log.cc:1198] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:16.607053 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80152844286545c68c7656ef9bbe8604"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53824
I20250110 03:07:16.607121 21467 raft_consensus.cc:604] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:16.607431 24966 tablet_copy_client.cc:806] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.607530 24966 tablet_copy_client.cc:670] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.607916 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.608130 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.608731 23100 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.608748 23363 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.608901 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.609025 24123 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.609385 24966 tablet_copy_client.cc:538] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.609645 23233 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.609879 25753 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.609982 25753 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.610108 24966 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.610908 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.611135 23363 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.612591 21194 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.612897 23233 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.614728 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.615289 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:16.615480 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.615595 25197 tablet_replica.cc:331] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.615664 25197 raft_consensus.cc:2238] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.615710 23363 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.615762 25197 raft_consensus.cc:2267] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.615869 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.616242 25197 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.616387 25754 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.617532 25197 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:16.617533 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.617579 25197 log.cc:1198] T 41fa30056c0a4b30ab8fcaa14645e7a4 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:16.617604 24966 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.617672 23363 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.617777 24966 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.617870 24966 ts_tablet_manager.cc:1397] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250110 03:07:16.617974 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.617981 24966 raft_consensus.cc:357] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.618053 24966 raft_consensus.cc:738] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.618086 24966 consensus_queue.cc:260] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.618188 24966 ts_tablet_manager.cc:1428] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.618398 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-4e11143dad6b402fb0d16c48285646c9 received from {username='slave'} at 127.14.19.134:56637
I20250110 03:07:16.618446 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-4e11143dad6b402fb0d16c48285646c9 on tablet 4e11143dad6b402fb0d16c48285646c9 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.618465 23100 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.618674 24122 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.618909 21187 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.619174 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:16.619614 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:16.619640 21181 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with cas_config_opid_index 7: aborting the task: latest config opid_index 12; task opid_index 7
I20250110 03:07:16.619720 25796 tablet_replica.cc:331] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.619777 25796 raft_consensus.cc:2238] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.619844 25796 raft_consensus.cc:2267] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.620144 25796 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.620920 24122 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.621479 25796 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:16.621532 25796 log.cc:1198] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:16.621932 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.622395 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.622373 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.622447 21195 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.622591 23233 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.623759 25753 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.623914 25753 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.624380 23233 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.624591 22561 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.625144 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.625273 23102 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.625303 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.625666 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.625766 22971 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.625811 25754 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.626819 23233 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.626974 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.627074 21195 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.627043 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.627192 24122 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.627295 24122 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.627367 24122 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.627780 21467 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:16.627764 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.628479 22971 raft_consensus.cc:1212] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.628796 23102 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250110 03:07:16.628927 21405 consensus_peers.cc:487] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.629357 23102 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.629726 24065 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.630252 22561 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.630185 23102 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.630293 23363 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.630389 22561 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.630496 24065 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.631407 23233 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.631601 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.631606 23101 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.632073 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.632258 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.633147 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.633306 24122 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.633298 22971 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.633843 24122 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.634020 24065 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.635078 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.635234 23364 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.635280 23233 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.636199 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.636534 21187 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 9 to 10, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.637104 21194 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 9 to 10, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.637945 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.638474 23233 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.638475 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.638624 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.638864 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.638937 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.639364 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.640004 25754 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.640491 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.640954 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.641005 23101 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.641077 21188 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.641172 23233 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.641678 22969 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.641813 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.642294 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.642406 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.642696 22968 raft_consensus.cc:867] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
W20250110 03:07:16.642740 22968 raft_consensus.cc:1574] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.642784 21337 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.642839 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.642887 21335 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.643029 21336 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.643291 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.643687 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:16.643782 25129 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:16.643824 25129 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.643754 21336 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.643791 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.643869 25129 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.644167 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.644260 23233 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.644305 25754 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.644402 24437 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.644644 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.644800 24437 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.644831 21336 raft_consensus.cc:867] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
W20250110 03:07:16.644865 21336 raft_consensus.cc:1574] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.644904 25754 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.644961 21336 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.645169 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.645911 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.646018 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.646057 25129 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.646009 23233 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.646692 21187 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.647388 25129 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:16.647440 25129 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:16.647505 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.647603 25725 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.647612 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.647662 25725 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.647740 25725 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.647810 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.648157 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.648273 23233 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.648379 24437 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.648617 22969 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:16.648800 24437 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.648890 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:16.648967 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20250110 03:07:16.649258 26446 consensus_queue.cc:1159] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" status { last_received { term: 1 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 1 index: 9 } }
I20250110 03:07:16.649358 22968 raft_consensus.cc:867] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
W20250110 03:07:16.649415 22968 raft_consensus.cc:1574] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 12' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.649456 24437 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.649497 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.649652 25725 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.650272 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.650415 23233 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.650468 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.650856 25725 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:16.650918 25725 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:16.650918 22968 raft_consensus.cc:867] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
W20250110 03:07:16.650959 22968 raft_consensus.cc:1574] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 13' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.651018 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.651449 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.651638 21190 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 12 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.652150 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.652804 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.652896 25921 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.652954 25921 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.653062 25921 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.653370 25921 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.653642 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.653982 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:16.654004 23363 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:16.654151 24122 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:16.654220 24123 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:16.654652 25921 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250110 03:07:16.654706 25921 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:16.654922 24123 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.655066 23363 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.655146 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.655824 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.656064 21187 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.656697 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:16.656788 25796 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.656848 25796 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.656910 25796 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.657006 22969 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.657213 25796 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.658483 25796 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:16.658530 25796 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:16.659832 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.662745 23363 raft_consensus.cc:1212] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.662778 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.663441 24965 ts_tablet_manager.cc:927] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:16.663478 22969 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:16.663568 24965 tablet_copy_client.cc:323] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:16.663753 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 4e11143dad6b402fb0d16c48285646c9 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:51125)
I20250110 03:07:16.663801 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 4e11143dad6b402fb0d16c48285646c9 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:51125: session id = e4d83809ce4746d5ad65e8462857fd9c-4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:16.663816 21730 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:16.663833 24123 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:16.663954 24123 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:16.664296 21751 tablet_copy_source_session.cc:215] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.664584 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e11143dad6b402fb0d16c48285646c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.665041 24123 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.665174 22969 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.665973 24965 tablet_copy_client.cc:806] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.666088 24965 tablet_copy_client.cc:670] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.666230 21730 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.666966 24965 tablet_copy_client.cc:538] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.667357 21187 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 13 to 14, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.667620 24965 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.667757 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.668160 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.668247 25197 tablet_replica.cc:331] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.668298 25197 raft_consensus.cc:2238] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.668465 25197 raft_consensus.cc:2267] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.668874 25197 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.670387 25197 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:16.670430 25197 log.cc:1198] T 41db68f089644324af518422d2cad2fc P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/41db68f089644324af518422d2cad2fc
I20250110 03:07:16.671131 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 41db68f089644324af518422d2cad2fc (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:16.672029 24965 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.672233 24965 tablet_bootstrap.cc:492] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.672322 24965 ts_tablet_manager.cc:1397] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250110 03:07:16.674639 23233 raft_consensus.cc:1212] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:07:16.674902 24965 raft_consensus.cc:357] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.675042 24965 raft_consensus.cc:738] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.675086 24965 consensus_queue.cc:260] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.675436 24965 ts_tablet_manager.cc:1428] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:16.675750 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-4e11143dad6b402fb0d16c48285646c9 received from {username='slave'} at 127.14.19.133:51125
I20250110 03:07:16.675801 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-4e11143dad6b402fb0d16c48285646c9 on tablet 4e11143dad6b402fb0d16c48285646c9 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.684878 24138 raft_consensus.cc:1059] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.685052 24138 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.685675 21336 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.685907 23100 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.686113 23363 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.686106 24165 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.686147 21729 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.686164 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.686344 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.686455 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.687171 24165 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.688369 23100 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.688369 21336 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.689114 21195 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.689813 21729 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.690361 21598 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.690999 21729 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.691087 23100 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.691308 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.691839 24122 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.692302 24123 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.692368 23364 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.692672 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.692813 23100 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.693461 21729 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.693918 24966 ts_tablet_manager.cc:927] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.694046 24966 tablet_copy_client.cc:323] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.694240 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 3b266cc7c4f341f785152932febcb79b from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:16.694293 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 3b266cc7c4f341f785152932febcb79b from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-3b266cc7c4f341f785152932febcb79b
I20250110 03:07:16.694953 21489 tablet_copy_source_session.cc:215] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.695274 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.695706 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.695741 23362 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.695920 21188 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.696552 23362 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.696846 24966 tablet_copy_client.cc:806] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.696974 24966 tablet_copy_client.cc:670] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.697351 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.697440 25725 tablet_replica.cc:331] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.697500 25725 raft_consensus.cc:2238] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.697568 25725 raft_consensus.cc:2267] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.697925 25725 ts_tablet_manager.cc:1905] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.698081 24123 raft_consensus.cc:1059] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.698189 24123 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.698527 21729 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.698770 23362 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.698818 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.699023 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.699321 23100 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.699458 25725 ts_tablet_manager.cc:1918] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.699520 25725 log.cc:1198] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:16.699559 24122 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.699968 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:16.700173 24966 tablet_copy_client.cc:538] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.700549 24122 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.700865 24966 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.701529 21729 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.702204 21187 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.702877 23362 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.703517 23100 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.703719 24203 raft_consensus.cc:1059] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.703923 24203 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.704597 23233 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.704633 21336 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.704664 22969 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.704807 23100 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.704882 24221 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.704982 24221 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.705061 24221 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.705204 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.706055 21598 consensus_queue.cc:237] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.706459 23362 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.706665 22561 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.706838 22874 raft_consensus.cc:1022] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer e4d83809ce4746d5ad65e8462857fd9c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.707166 21729 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.707160 23100 raft_consensus.cc:1270] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.707340 24123 consensus_queue.cc:1035] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.707427 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.707974 24221 raft_consensus.cc:1059] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.707962 24123 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.708123 24221 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.708570 23362 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.708823 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.709071 21730 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.709048 21195 catalog_manager.cc:5526] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.709371 22969 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.709408 23100 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.710074 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.710122 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.710212 25921 tablet_replica.cc:331] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.710265 25921 raft_consensus.cc:2238] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.710361 25921 raft_consensus.cc:2267] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.710623 23232 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.710767 25921 ts_tablet_manager.cc:1905] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.710873 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.711227 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.711347 21598 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.711813 23364 raft_consensus.cc:2949] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.712121 25921 ts_tablet_manager.cc:1918] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.712177 25921 log.cc:1198] T 22fe8e3aabc64a848ffc1e4f21fa1b51 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:16.712512 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 22fe8e3aabc64a848ffc1e4f21fa1b51 (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:16.713042 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.713140 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.714025 22874 raft_consensus.cc:1022] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.714078 22874 raft_consensus.cc:1059] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.714201 22874 consensus_queue.cc:237] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.714607 23364 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.714797 22884 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.714968 21730 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.715112 23232 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.715333 22884 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.715404 24221 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.716142 24221 raft_consensus.cc:1022] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.716465 24221 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.716583 21729 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.717509 22968 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.717562 23100 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.718286 21336 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.718482 24966 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.718492 23233 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.718892 21195 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.720360 22874 raft_consensus.cc:1059] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.720518 22874 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.721247 23100 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.721546 21597 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.721567 21729 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.721786 21336 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.721777 24065 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.721890 24065 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.723034 24203 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.723398 23233 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.723595 24203 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.724295 24221 raft_consensus.cc:1022] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.724649 24203 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.725648 23233 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.726071 21730 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.726760 24065 raft_consensus.cc:1059] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.726891 24065 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.727423 23233 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.727510 25088 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.727527 23100 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.727721 22971 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.727712 21188 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER, now has a pending config: VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132), VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130), VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129), VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135), VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134), VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.728111 22968 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.728255 23101 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.728516 24966 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.728520 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.728663 24966 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20250110 03:07:16.728577 21336 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.728783 24966 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.728863 24966 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.728893 24966 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.729018 24966 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.729192 21489 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-3b266cc7c4f341f785152932febcb79b received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:16.729234 21489 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-3b266cc7c4f341f785152932febcb79b on tablet 3b266cc7c4f341f785152932febcb79b with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.729573 21335 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.729951 21730 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.730235 24203 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.730433 23231 raft_consensus.cc:1212] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.730561 23363 raft_consensus.cc:1212] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.731351 22874 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.731448 22874 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.731508 22874 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.732134 22968 raft_consensus.cc:867] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
W20250110 03:07:16.732223 22968 raft_consensus.cc:1574] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.732326 22968 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.733017 22874 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.733214 21598 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.733515 24203 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.734395 21729 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.734745 23232 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.734730 21195 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.734997 23101 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.735847 23231 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.736332 25035 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.736444 25035 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.736320 21335 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.736631 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:36717)
I20250110 03:07:16.736678 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:36717: session id = b2e7bce763e24b63bc760b0b82b53454-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:16.736877 25088 raft_consensus.cc:1059] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.737031 25088 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.737196 21358 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.737742 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.738320 23231 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.738400 21187 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132), VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130), VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129), VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135), VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134), VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.738518 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.739032 25035 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.739048 24168 raft_consensus.cc:988] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: : Instructing follower c57c9dfa476f4d21b79f81114c5d74cb to start an election
I20250110 03:07:16.739094 24168 raft_consensus.cc:1076] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Signalling peer c57c9dfa476f4d21b79f81114c5d74cb to start an election
I20250110 03:07:16.739152 25035 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.739236 21600 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.739277 21598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
from {username='slave'} at 127.14.19.129:51055
I20250110 03:07:16.739351 23363 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.739341 21598 raft_consensus.cc:491] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:16.739387 21598 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.739576 22968 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.739728 24065 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.739827 24065 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.739905 24065 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.740087 21598 raft_consensus.cc:513] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.740204 21598 leader_election.cc:290] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 2 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:16.740291 21729 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.740458 24220 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.740677 23363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
I20250110 03:07:16.740770 23363 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.740836 21728 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.740932 23364 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.740984 23362 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.741422 23363 raft_consensus.cc:2463] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 2.
I20250110 03:07:16.741789 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cc5781db40c4148ba1889c45405efc2" candidate_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:16.741864 21335 raft_consensus.cc:3049] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:16.741891 21335 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:16.741946 21335 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.742033 21335 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.742025 21533 leader_election.cc:304] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c57c9dfa476f4d21b79f81114c5d74cb, d0be0af85dee41dd8d01ef4b2af7726f; no voters:
I20250110 03:07:16.742106 24123 raft_consensus.cc:2798] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:16.742142 24123 raft_consensus.cc:695] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Becoming Leader. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:16.742139 21194 catalog_manager.cc:5526] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.742182 24123 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.742648 21467 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.742966 21190 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: term changed from 1 to 2, leader changed from a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) to c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131). New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.743471 24220 raft_consensus.cc:1022] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.743578 23232 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.743760 24203 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.744004 21467 consensus_queue.cc:237] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.744446 21729 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.744606 24220 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.744643 23364 raft_consensus.cc:1212] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.745648 26668 ts_tablet_manager.cc:927] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.745664 21335 raft_consensus.cc:2463] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c57c9dfa476f4d21b79f81114c5d74cb in term 2.
I20250110 03:07:16.745874 26668 tablet_copy_client.cc:323] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.746129 24966 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.746255 24966 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.746435 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:38517)
I20250110 03:07:16.746493 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:38517: session id = 531cf07bcce6435c9e18185c60b2118d-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:16.746732 25035 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.747017 21358 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.747352 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.747752 25035 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.748577 24966 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.748709 24966 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.748894 22874 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.749950 21730 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.750062 24966 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.750741 24966 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:16.751158 23232 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.752090 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet b5064ab85103488f873fffef55fd78fd from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:16.752136 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet b5064ab85103488f873fffef55fd78fd from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-b5064ab85103488f873fffef55fd78fd
I20250110 03:07:16.752692 21489 tablet_copy_source_session.cc:215] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.752885 21188 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.753090 23364 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.754048 26668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.754251 23101 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.754462 24221 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.754590 24221 raft_consensus.cc:1059] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.754622 22967 raft_consensus.cc:1212] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.754710 24221 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.754878 22966 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:16.755060 26668 tablet_copy_client.cc:806] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.755112 21730 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.755156 26668 tablet_copy_client.cc:670] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.755265 22874 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.755443 21599 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.755456 23232 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.755543 21600 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.755959 24065 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.756028 22874 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.756784 23362 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.756937 23101 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.757040 21335 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.757182 22884 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.757368 23232 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.758230 24065 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.758193 21187 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.758481 22970 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.758865 25088 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.759086 23364 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.759034 25088 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.759164 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d52b50e3c87b47bead3a25f7255d1c42 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.759030 24203 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.759385 24203 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.759838 22970 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.759865 21599 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.760226 21730 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.761056 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.761178 25921 tablet_replica.cc:331] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.761238 25921 raft_consensus.cc:2238] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.761318 25921 pending_rounds.cc:62] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Trying to abort 1 pending ops.
I20250110 03:07:16.761260 21335 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.761443 21599 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:16.761390 25921 pending_rounds.cc:69] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7112615676833058816 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d52b50e3c87b47bead3a25f7255d1c42" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } } }
I20250110 03:07:16.761538 25921 raft_consensus.cc:2911] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20250110 03:07:16.761574 25921 raft_consensus.cc:2883] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:16.761597 25921 raft_consensus.cc:2267] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.761626 25102 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.762233 23232 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.762249 21195 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.762315 23231 raft_consensus.cc:1212] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.762974 23364 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.763619 25921 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.763971 21468 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.763971 21467 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.764940 22970 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.764986 23101 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.765025 25921 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:16.765070 25921 log.cc:1198] T d52b50e3c87b47bead3a25f7255d1c42 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:16.765117 21599 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.765314 25089 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.765436 25089 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.765506 25089 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.765868 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet d52b50e3c87b47bead3a25f7255d1c42 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.766142 26668 tablet_copy_client.cc:538] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.766172 23231 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.766336 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.766429 24065 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.766537 23363 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.766636 23231 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.766870 26668 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.767277 24065 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.767333 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d52b50e3c87b47bead3a25f7255d1c42 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.767537 23101 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.767668 22970 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.767740 23232 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.768023 24221 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.757385 25035 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.768453 21187 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 10 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.768541 25035 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.768836 24221 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.769735 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.769842 25725 tablet_replica.cc:331] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.769904 25725 raft_consensus.cc:2238] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.769863 21599 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.770002 25725 raft_consensus.cc:2267] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.770295 25725 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.770656 23230 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.771638 25725 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:16.771683 25725 log.cc:1198] T d52b50e3c87b47bead3a25f7255d1c42 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:16.771843 25088 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.771875 23362 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.772002 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet d52b50e3c87b47bead3a25f7255d1c42 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.772565 23232 raft_consensus.cc:1212] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.772945 21730 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.773233 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 122762e845a046f4a84afeeacc01b42f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.773304 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a6000ace3863451db3fdd8d150cd0944 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.773365 21598 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.773545 23362 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.773638 24065 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.773866 21598 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.773903 21190 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.774219 24966 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.774362 25035 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250110 03:07:16.774493 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 122762e845a046f4a84afeeacc01b42f with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.774497 25035 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.774608 25035 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.774645 25035 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.774739 21598 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.774946 25921 tablet_replica.cc:331] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.774972 25035 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.775007 25921 raft_consensus.cc:2238] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.775094 25921 raft_consensus.cc:2267] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.775132 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.135:36717
I20250110 03:07:16.775171 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.775374 21468 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.775403 21195 catalog_manager.cc:5526] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 6 to 7, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.775820 23101 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.775985 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.776042 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet a6000ace3863451db3fdd8d150cd0944 with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:16.776125 25796 tablet_replica.cc:331] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.776136 25921 ts_tablet_manager.cc:1905] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.776170 25796 raft_consensus.cc:2238] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:16.776201 25796 pending_rounds.cc:62] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Trying to abort 1 pending ops.
I20250110 03:07:16.776276 22965 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.776252 25796 pending_rounds.cc:69] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7112615676780982272 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a6000ace3863451db3fdd8d150cd0944" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } } }
I20250110 03:07:16.776356 25796 raft_consensus.cc:2911] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20250110 03:07:16.776382 25796 raft_consensus.cc:2883] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:16.776400 25796 raft_consensus.cc:2267] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:16.776526 22884 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.776748 25089 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.777027 21468 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.777261 23101 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.777331 22965 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.777696 25921 ts_tablet_manager.cc:1918] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:16.777745 25921 log.cc:1198] T 122762e845a046f4a84afeeacc01b42f P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/122762e845a046f4a84afeeacc01b42f
I20250110 03:07:16.778057 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 122762e845a046f4a84afeeacc01b42f (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.778211 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d52b50e3c87b47bead3a25f7255d1c42 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.778229 23362 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.778484 24220 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.778893 21467 consensus_queue.cc:237] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.778466 21195 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.779371 24966 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:16.779474 24966 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
I20250110 03:07:16.779541 23232 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.779598 21730 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.779603 24966 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.779647 23362 raft_consensus.cc:1270] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.779676 24966 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:16.779712 24966 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.779759 24221 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.779783 22884 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.779814 24966 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.779847 22884 consensus_queue.cc:1035] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.780009 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.134:38517
W20250110 03:07:16.779996 26659 consensus_queue.cc:1159] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b2e7bce763e24b63bc760b0b82b53454" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20250110 03:07:16.780043 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:16.780207 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:16.780403 24065 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.780510 25796 ts_tablet_manager.cc:1905] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.780632 23362 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.781571 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a6000ace3863451db3fdd8d150cd0944 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.781821 25796 ts_tablet_manager.cc:1918] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:07:16.781797 21187 catalog_manager.cc:5526] T a6000ace3863451db3fdd8d150cd0944 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.781864 25796 log.cc:1198] T a6000ace3863451db3fdd8d150cd0944 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:16.782130 25796 tablet_replica.cc:331] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:16.782179 25796 raft_consensus.cc:2238] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.782235 25796 raft_consensus.cc:2267] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.782527 22971 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.782500 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet a6000ace3863451db3fdd8d150cd0944 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:16.782712 25793 tablet_replica.cc:331] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:16.782769 25793 raft_consensus.cc:2238] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.782766 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet a6000ace3863451db3fdd8d150cd0944 (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:16.782807 22965 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.782833 25793 raft_consensus.cc:2267] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.783036 22968 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.783057 22874 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.783125 24221 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.783195 22967 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.783234 21730 raft_consensus.cc:2949] T a6000ace3863451db3fdd8d150cd0944 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.783368 23232 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.783634 22967 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.784173 25796 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.784286 23101 raft_consensus.cc:1212] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.784536 22967 raft_consensus.cc:867] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
W20250110 03:07:16.784572 22967 raft_consensus.cc:1574] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 2 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.784644 22967 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.785351 25796 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:16.785395 25796 log.cc:1198] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:16.785435 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 122762e845a046f4a84afeeacc01b42f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.785705 23362 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.785800 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet d52b50e3c87b47bead3a25f7255d1c42 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:16.785777 21196 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.786396 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 122762e845a046f4a84afeeacc01b42f with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:16.787271 24123 raft_consensus.cc:1059] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.787420 24123 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.787959 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:16.788048 21467 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:16.788224 24438 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:16.788389 24138 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:16.788851 21467 consensus_queue.cc:237] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.789319 23362 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:16.789489 24138 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:16.790045 23232 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.790069 23363 raft_consensus.cc:1270] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:16.790216 25089 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.790264 25793 ts_tablet_manager.cc:1905] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.790283 24065 consensus_queue.cc:1035] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:16.790786 22884 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.790877 23232 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.791618 25793 ts_tablet_manager.cc:1918] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:16.791664 25793 log.cc:1198] T a6000ace3863451db3fdd8d150cd0944 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:16.791913 25793 tablet_replica.cc:331] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:16.791965 25793 raft_consensus.cc:2238] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.792057 25793 raft_consensus.cc:2267] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.792109 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet a6000ace3863451db3fdd8d150cd0944 (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:16.792271 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 122762e845a046f4a84afeeacc01b42f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.792321 25793 ts_tablet_manager.cc:1905] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.792649 23363 raft_consensus.cc:2949] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.792906 26668 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.793139 26668 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.793229 26668 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250110 03:07:16.793334 26668 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.793419 26668 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.793460 26668 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:16.793526 25793 ts_tablet_manager.cc:1918] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:16.793567 25793 log.cc:1198] T 122762e845a046f4a84afeeacc01b42f P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/122762e845a046f4a84afeeacc01b42f
I20250110 03:07:16.793571 25035 ts_tablet_manager.cc:927] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:16.793700 25035 tablet_copy_client.cc:323] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:16.793947 26668 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:16.794025 21190 catalog_manager.cc:5526] T 122762e845a046f4a84afeeacc01b42f P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.794040 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 122762e845a046f4a84afeeacc01b42f (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.794493 21489 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet a790eedb619d47c6a3ba7eba194b70c1 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:16.794534 21489 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet a790eedb619d47c6a3ba7eba194b70c1 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:16.795166 21489 tablet_copy_source_session.cc:215] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.795264 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-b5064ab85103488f873fffef55fd78fd received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:16.795296 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-b5064ab85103488f873fffef55fd78fd on tablet b5064ab85103488f873fffef55fd78fd with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.795692 24138 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.796330 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a790eedb619d47c6a3ba7eba194b70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.796674 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 122762e845a046f4a84afeeacc01b42f with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:16.796849 25867 tablet_replica.cc:331] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:16.796897 25867 raft_consensus.cc:2238] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.796927 25867 raft_consensus.cc:2267] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.797034 21468 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.797376 25035 tablet_copy_client.cc:806] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.797490 25035 tablet_copy_client.cc:670] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.798137 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.798223 23362 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.798296 21187 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 4 to 5, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.798859 25867 ts_tablet_manager.cc:1905] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.800238 25867 ts_tablet_manager.cc:1918] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.800293 25867 log.cc:1198] T 122762e845a046f4a84afeeacc01b42f P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/122762e845a046f4a84afeeacc01b42f
I20250110 03:07:16.801038 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 122762e845a046f4a84afeeacc01b42f (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:16.801154 21598 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.801988 23362 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.801990 25035 tablet_copy_client.cc:538] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.802803 25035 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.804342 21468 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.804316 24437 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.804525 24437 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.805006 24437 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.806303 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:16.806340 21468 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.806334 21187 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 5 to 6, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.806517 25102 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.806568 23362 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.806874 25035 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.807054 25035 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.807269 25035 ts_tablet_manager.cc:1397] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250110 03:07:16.807572 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c5d0922741634918830ee2c163cbce24 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.807586 25035 raft_consensus.cc:357] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.807667 25035 raft_consensus.cc:738] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.807705 25035 consensus_queue.cc:260] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.807749 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d0922741634918830ee2c163cbce24 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.807827 25725 tablet_replica.cc:331] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.807833 25035 ts_tablet_manager.cc:1428] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.807881 25725 raft_consensus.cc:2238] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.807933 25725 raft_consensus.cc:2267] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.808079 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-a790eedb619d47c6a3ba7eba194b70c1 received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:16.808122 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-a790eedb619d47c6a3ba7eba194b70c1 on tablet a790eedb619d47c6a3ba7eba194b70c1 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.808207 25725 ts_tablet_manager.cc:1905] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.809482 25725 ts_tablet_manager.cc:1918] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:16.809535 25725 log.cc:1198] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/c5d0922741634918830ee2c163cbce24
I20250110 03:07:16.810024 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet c5d0922741634918830ee2c163cbce24 (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:16.814117 21335 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:16.814677 24665 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.815047 23362 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.815516 21598 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:07:16.815587 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:07:16.815738 24665 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.815838 24665 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.816327 25102 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.816541 21598 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.817404 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.817929 21190 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 9 to 11, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.818219 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.818650 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:16.818779 25725 tablet_replica.cc:331] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:16.818845 25725 raft_consensus.cc:2238] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.818889 25725 raft_consensus.cc:2267] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.819223 25725 ts_tablet_manager.cc:1905] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.820454 25725 ts_tablet_manager.cc:1918] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20250110 03:07:16.820513 25725 log.cc:1198] T 44bb3816ff5a4d44a249b8bbabdc7b04 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:16.820888 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 44bb3816ff5a4d44a249b8bbabdc7b04 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:16.829272 24726 raft_consensus.cc:1059] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.829926 24726 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.830623 21730 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.831143 23232 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.831375 26778 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.831744 23101 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.832082 22967 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.832280 21468 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.832686 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.832798 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.833902 24168 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.834017 24168 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.838045 25693 raft_consensus.cc:1059] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.838222 25693 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.838519 23362 raft_consensus.cc:1212] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.839112 21335 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:16.839181 22966 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:16.839390 25692 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:16.839803 25692 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:16.840097 21467 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:16.840651 25692 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:16.841611 24152 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.842018 25692 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.842891 21730 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.842945 21196 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.843837 22966 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.844540 22967 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.845116 21187 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 4 to 5, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.845871 25089 raft_consensus.cc:1059] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:16.846063 25089 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.846638 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.846719 24152 raft_consensus.cc:1059] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.846796 21598 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.846906 23233 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.846890 24152 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.846997 24221 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.847326 24203 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.847432 24203 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.847622 21468 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.847945 23101 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.848052 25174 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.848176 24726 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.848309 21468 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.848609 23101 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.849228 22966 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.849829 21729 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.849999 24221 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.850256 24726 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.851018 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.851409 21467 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.852238 21335 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.852452 24203 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.853230 24726 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.853494 21728 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.854462 23101 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.854821 22967 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.854984 25174 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.855252 21730 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:16.855741 22968 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.855680 21190 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.855994 22967 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.856175 25693 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.857235 24221 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.857352 21336 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.858325 21335 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.858289 23362 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.858289 21598 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.858740 21730 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.858959 25176 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.859477 21187 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.859803 23101 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.860031 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.860428 25693 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.860554 25693 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.861639 24152 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.862104 21335 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.862618 23100 raft_consensus.cc:1212] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:16.862582 23101 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.862995 22966 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:16.863790 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9b16001c451b455f9eda42f3c98a2fdd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.863842 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 58e33e63f48f4b758c19774888345ee0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.864038 21190 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 5 to 6, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.864110 25088 raft_consensus.cc:1059] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.864250 25088 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.864400 21194 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.864707 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.865147 22965 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.865371 25089 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.865446 24203 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.865602 22966 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.866431 22968 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.866530 21599 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.866693 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b16001c451b455f9eda42f3c98a2fdd with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.866793 26778 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.866830 25197 tablet_replica.cc:331] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.866888 25197 raft_consensus.cc:2238] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.866958 25197 raft_consensus.cc:2267] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.867101 22967 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.867244 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e33e63f48f4b758c19774888345ee0 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.867825 21728 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.868137 25197 ts_tablet_manager.cc:1905] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.868170 23231 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.868376 23233 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.868662 26778 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.869091 24203 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.869148 23233 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.869539 25197 ts_tablet_manager.cc:1918] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:16.869581 25197 log.cc:1198] T 9b16001c451b455f9eda42f3c98a2fdd P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:16.869859 25197 tablet_replica.cc:331] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.869932 25197 raft_consensus.cc:2238] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.869936 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 9b16001c451b455f9eda42f3c98a2fdd (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:16.869997 25197 pending_rounds.cc:62] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Trying to abort 1 pending ops.
I20250110 03:07:16.870045 25197 pending_rounds.cc:69] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7112615677324558336 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "58e33e63f48f4b758c19774888345ee0" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } } }
I20250110 03:07:16.870146 25197 raft_consensus.cc:2911] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250110 03:07:16.870172 25197 raft_consensus.cc:2883] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:16.870191 25197 raft_consensus.cc:2267] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.870180 23233 raft_consensus.cc:867] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
W20250110 03:07:16.870235 23233 raft_consensus.cc:1574] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.870301 23233 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.870959 25089 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.872138 25174 raft_consensus.cc:1059] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.872305 25174 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.872478 23233 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.872545 21195 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.872849 21728 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.873028 23362 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.873081 24726 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.873227 23101 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.873462 25176 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.873929 25176 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.874704 22965 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.875171 21728 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.876420 21599 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.876570 24965 ts_tablet_manager.cc:927] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:16.876535 23101 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.876698 24965 tablet_copy_client.cc:323] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:16.876864 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:16.876907 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:16.877522 21620 tablet_copy_source_session.cc:215] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.877710 21188 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.877956 24965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 03:07:16.878216 26790 consensus_queue.cc:1159] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab3ed3d8c2394f78943461e22853c7ac" status { last_received { term: 1 index: 8 } last_committed_idx: 7 last_received_current_leader { term: 1 index: 8 } }
I20250110 03:07:16.878759 25197 ts_tablet_manager.cc:1905] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.879108 23231 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.879117 22968 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.879576 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.880136 25035 ts_tablet_manager.cc:927] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:16.880172 24726 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.880272 25035 tablet_copy_client.cc:323] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:16.880302 24726 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.880479 21468 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.880678 22968 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.880746 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 62aacc8deebd43debd76a2c10815da2f from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:36717)
I20250110 03:07:16.880784 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 62aacc8deebd43debd76a2c10815da2f from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:36717: session id = b2e7bce763e24b63bc760b0b82b53454-62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:16.880795 23231 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.881079 23229 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.881218 21358 tablet_copy_source_session.cc:215] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:16.881685 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.881855 25035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:16.881861 24065 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.882025 21181 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet 58e33e63f48f4b758c19774888345ee0 with cas_config_opid_index 7: aborting the task: latest config opid_index 10; task opid_index 7
I20250110 03:07:16.882215 21730 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.882511 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.882762 25197 ts_tablet_manager.cc:1918] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.882824 25197 log.cc:1198] T 58e33e63f48f4b758c19774888345ee0 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/58e33e63f48f4b758c19774888345ee0
I20250110 03:07:16.883157 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 58e33e63f48f4b758c19774888345ee0 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:16.883281 21730 raft_consensus.cc:867] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.883296 25035 tablet_copy_client.cc:806] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
W20250110 03:07:16.883318 21730 raft_consensus.cc:1574] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.883404 25035 tablet_copy_client.cc:670] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.883399 21730 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.884059 24965 tablet_copy_client.cc:806] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:16.884059 24065 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.884188 24965 tablet_copy_client.cc:670] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:16.884523 23231 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.884697 24220 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.885002 22968 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.885177 23231 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.886132 23231 raft_consensus.cc:867] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
W20250110 03:07:16.886176 23231 raft_consensus.cc:1574] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.886272 23231 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.887683 24220 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.888356 24203 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.888813 22968 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.889659 21730 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.889853 23362 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.891113 21196 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.891512 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0760e9b8e3c642f09c3287ca586a2a1d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.892038 24221 raft_consensus.cc:1059] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.892153 24221 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.892566 22968 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.892596 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.892771 22874 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.892874 22874 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.893177 23229 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.893358 23231 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.893512 22884 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.894061 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:16.894061 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:16.894155 25793 tablet_replica.cc:331] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:16.894219 25793 raft_consensus.cc:2238] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.894222 24203 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.894260 24065 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:16.894320 25793 raft_consensus.cc:2267] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.894860 21730 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.895705 25793 ts_tablet_manager.cc:1905] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.896324 22968 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.896943 23229 raft_consensus.cc:1212] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.10] Dedup: 1.9->[1.10-1.10]
I20250110 03:07:16.897053 25793 ts_tablet_manager.cc:1918] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.897099 25793 log.cc:1198] T 0760e9b8e3c642f09c3287ca586a2a1d P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/0760e9b8e3c642f09c3287ca586a2a1d
W20250110 03:07:16.897399 21182 catalog_manager.cc:5049] ChangeConfig:REMOVE_PEER RPC for tablet 0760e9b8e3c642f09c3287ca586a2a1d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 9 but the committed config has opid_index of 10
I20250110 03:07:16.897466 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 0760e9b8e3c642f09c3287ca586a2a1d (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:16.897399 23362 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.897732 21195 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.898759 23229 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.900118 21468 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.900344 25035 tablet_copy_client.cc:538] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.900779 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.900939 24221 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.901078 25035 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:16.901345 23229 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.901610 24965 tablet_copy_client.cc:538] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:16.901588 24221 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.902294 24965 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:16.902931 22968 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:16.903088 24221 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:16.903203 24221 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.903303 23362 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.904197 21188 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.904323 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0760e9b8e3c642f09c3287ca586a2a1d with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.904448 23229 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.904855 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.904947 25921 tablet_replica.cc:331] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.905022 25921 raft_consensus.cc:2238] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.905114 25921 raft_consensus.cc:2267] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.905237 22968 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:16.905771 25921 ts_tablet_manager.cc:1905] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.907200 25921 ts_tablet_manager.cc:1918] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:16.907253 25921 log.cc:1198] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:16.907574 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 0760e9b8e3c642f09c3287ca586a2a1d (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:16.908804 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80152844286545c68c7656ef9bbe8604"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54014
I20250110 03:07:16.908869 21467 raft_consensus.cc:604] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:16.909564 24965 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.909766 24965 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:16.909834 24965 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250110 03:07:16.909916 24965 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.909958 24965 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:16.909977 24965 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:16.910046 24965 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:16.910256 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-81b3bd116ad54a9fa10d2d0c829bb467 received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:16.910312 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-81b3bd116ad54a9fa10d2d0c829bb467 on tablet 81b3bd116ad54a9fa10d2d0c829bb467 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:16.915119 25035 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:16.915524 23101 raft_consensus.cc:1212] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250110 03:07:16.915742 24220 raft_consensus.cc:988] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: : Instructing follower 9da9d601b7d841b5a8f2c2c53f62a951 to start an election
I20250110 03:07:16.915798 24220 raft_consensus.cc:1076] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Signalling peer 9da9d601b7d841b5a8f2c2c53f62a951 to start an election
I20250110 03:07:16.915865 25035 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:16.915974 25035 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20250110 03:07:16.915987 21730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80152844286545c68c7656ef9bbe8604"
dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
from {username='slave'} at 127.14.19.130:58095
I20250110 03:07:16.916075 21730 raft_consensus.cc:491] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:16.916112 21730 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.916096 25035 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.916162 25035 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:16.916195 25035 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.916894 21730 raft_consensus.cc:513] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.917093 21730 leader_election.cc:290] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 2 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:16.917095 25035 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:16.917305 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-62aacc8deebd43debd76a2c10815da2f received from {username='slave'} at 127.14.19.135:36717
I20250110 03:07:16.917352 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-62aacc8deebd43debd76a2c10815da2f on tablet 62aacc8deebd43debd76a2c10815da2f with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:16.918490 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:16.918623 21599 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.918658 21467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:16.918763 21467 raft_consensus.cc:3049] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:16.918797 21467 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:16.918856 21467 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.918953 21467 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.919392 21599 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 2.
I20250110 03:07:16.919718 21467 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 2.
I20250110 03:07:16.920444 23362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80152844286545c68c7656ef9bbe8604" candidate_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
I20250110 03:07:16.920511 23362 raft_consensus.cc:3054] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:16.920564 21666 leader_election.cc:304] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, ab3ed3d8c2394f78943461e22853c7ac, c57c9dfa476f4d21b79f81114c5d74cb; no voters:
I20250110 03:07:16.920666 26797 raft_consensus.cc:2798] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:16.920712 26797 raft_consensus.cc:695] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEADER]: Becoming Leader. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Running, Role: LEADER
I20250110 03:07:16.920769 26797 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:16.921123 23362 raft_consensus.cc:2463] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9da9d601b7d841b5a8f2c2c53f62a951 in term 2.
I20250110 03:07:16.922359 21194 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: term changed from 1 to 2, leader changed from ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) to 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132). New cstate: current_term: 2 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.935165 22561 raft_consensus.cc:1059] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.935328 22561 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:16.935956 21730 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.936336 23229 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.936318 24438 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.936479 21467 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.936777 24165 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.936898 24165 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.936920 22968 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:16.937078 24438 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.939117 24438 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.940299 23101 raft_consensus.cc:1212] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.940949 21730 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.942168 22968 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.942185 21187 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.945006 24165 raft_consensus.cc:1059] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.945206 24165 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.946228 21730 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.946628 22968 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.946799 23231 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.946916 23229 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.947365 25753 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.948004 25753 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.948940 25753 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.950345 25753 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.951586 21187 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.951583 21729 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.953040 22968 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.955855 21335 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:16.956640 23231 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:16.956936 21274 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.957015 24152 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.957198 21469 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.957569 24152 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.957928 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.958276 24152 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.958429 24152 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.958904 21466 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.960268 21187 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 7 to 8, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.961198 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.962875 24726 raft_consensus.cc:1059] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:16.963037 24726 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
W20250110 03:07:16.963804 21274 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:16.964980 23231 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.965169 21466 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.965343 26778 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.965993 23230 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.966434 26778 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.967528 26816 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.968871 21194 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.969067 23230 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.970628 21466 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.971953 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.972236 21468 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.972572 24122 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:16.972908 21468 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:16.973799 21468 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.974833 24726 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.975404 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:16.977425 21599 raft_consensus.cc:1212] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:16.977465 23229 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.977923 21599 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.978874 23229 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.979079 24122 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.979521 22968 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.979728 24122 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.980502 21467 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.980759 24122 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:16.981712 24122 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.982913 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ecbd6afd612b4e86b94429633ac15082 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.983019 21187 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:16.983688 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet ecbd6afd612b4e86b94429633ac15082 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:16.983739 25089 raft_consensus.cc:1059] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:16.983819 21599 consensus_queue.cc:237] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:16.983882 25089 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.984012 25921 tablet_replica.cc:331] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:16.984083 25921 raft_consensus.cc:2238] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.984153 25921 raft_consensus.cc:2267] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.984294 23229 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.984402 23362 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.984586 25921 ts_tablet_manager.cc:1905] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.984736 24203 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.985349 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.985261 22968 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.985515 24203 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.985950 25921 ts_tablet_manager.cc:1918] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20250110 03:07:16.985993 25921 log.cc:1198] T ecbd6afd612b4e86b94429633ac15082 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:16.986303 22967 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.986325 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet ecbd6afd612b4e86b94429633ac15082 (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:16.986507 24203 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.986596 23101 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.986752 21599 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:16.986981 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.987017 22967 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.987088 24203 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:16.987183 24438 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.987761 23229 raft_consensus.cc:1270] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:16.987938 24438 consensus_queue.cc:1035] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:16.987984 24203 raft_consensus.cc:1022] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer e4d83809ce4746d5ad65e8462857fd9c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:16.988541 25753 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.989631 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ecbd6afd612b4e86b94429633ac15082 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:16.990015 22968 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.990136 21187 catalog_manager.cc:5526] T ecbd6afd612b4e86b94429633ac15082 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:16.991040 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet ecbd6afd612b4e86b94429633ac15082 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:16.991173 25197 tablet_replica.cc:331] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:16.991232 25197 raft_consensus.cc:2238] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:16.991292 25197 pending_rounds.cc:62] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Trying to abort 1 pending ops.
I20250110 03:07:16.991312 25197 pending_rounds.cc:69] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7112615677861175296 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ecbd6afd612b4e86b94429633ac15082" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } } }
I20250110 03:07:16.991473 25197 raft_consensus.cc:2911] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250110 03:07:16.991508 25197 raft_consensus.cc:2883] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:16.991526 25197 raft_consensus.cc:2267] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:16.991966 25197 ts_tablet_manager.cc:1905] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:16.993335 25197 ts_tablet_manager.cc:1918] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:16.993270 23229 raft_consensus.cc:2949] T ecbd6afd612b4e86b94429633ac15082 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:16.993396 25197 log.cc:1198] T ecbd6afd612b4e86b94429633ac15082 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:16.993737 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet ecbd6afd612b4e86b94429633ac15082 (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:16.994176 25088 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.995488 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.996194 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.996556 21187 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.997126 21599 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:16.998462 22874 raft_consensus.cc:1059] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:16.998656 22874 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:16.999666 23362 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:16.999884 23101 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.000046 22968 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.000243 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.000459 21599 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.000669 24220 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.000768 24220 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.000823 24220 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.000905 23100 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.001539 25089 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.001914 23362 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.002260 27030 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.003223 24220 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.003500 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.004190 21599 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.004789 24221 raft_consensus.cc:1059] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:17.004813 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.004895 24221 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.005571 23363 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.006024 21194 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.006124 27030 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.006264 23100 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.006804 23363 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.007457 23101 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.007735 25089 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.007869 21599 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.008934 24220 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.009147 22968 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.009366 24220 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.009557 24966 ts_tablet_manager.cc:927] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:17.009680 24966 tablet_copy_client.cc:323] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:17.009862 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:54949)
I20250110 03:07:17.009918 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:54949: session id = 531cf07bcce6435c9e18185c60b2118d-81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:17.010514 21620 tablet_copy_source_session.cc:215] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:17.010869 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.010874 24966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:17.011054 24220 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.011561 24220 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.011687 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.012188 24966 tablet_copy_client.cc:806] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:17.012319 24966 tablet_copy_client.cc:670] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:17.012753 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.012806 21599 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.013916 21196 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.013953 23101 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.016590 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.017122 23362 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.017372 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.017385 24966 tablet_copy_client.cc:538] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:17.017927 23363 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.018042 24966 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:17.018103 22968 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.018129 23101 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.018424 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.018517 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.018805 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.018880 25089 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.019865 25088 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.019979 23101 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.020270 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.021097 21190 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.021173 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49c3c5c20dcc4474b207b9c73904219f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.021425 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.021507 23363 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.022408 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c3c5c20dcc4474b207b9c73904219f with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:17.022500 25793 tablet_replica.cc:331] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:17.022578 25793 raft_consensus.cc:2238] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.022667 25793 raft_consensus.cc:2267] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.023015 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.023154 25793 ts_tablet_manager.cc:1905] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.023519 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.023680 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.023720 22968 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.023952 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.024484 25793 ts_tablet_manager.cc:1918] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:17.024523 25793 log.cc:1198] T 49c3c5c20dcc4474b207b9c73904219f P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:17.024953 23101 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.025079 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 49c3c5c20dcc4474b207b9c73904219f (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:17.025172 25089 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.025375 23363 raft_consensus.cc:1212] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250110 03:07:17.025305 22884 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.025475 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.026298 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.026527 23101 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.027858 21187 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 10 to 11, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.028978 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49c3c5c20dcc4474b207b9c73904219f with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.030078 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c3c5c20dcc4474b207b9c73904219f with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:17.030280 27068 tablet_replica.cc:331] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:17.030325 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.030603 27068 raft_consensus.cc:2238] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.030728 27068 raft_consensus.cc:2267] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.030797 22968 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.030910 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.031098 27068 ts_tablet_manager.cc:1905] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.031140 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.031235 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.031800 24221 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.031903 24966 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:17.032088 24966 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:17.032203 24966 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250110 03:07:17.032338 27068 ts_tablet_manager.cc:1918] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:17.032410 27068 log.cc:1198] T 49c3c5c20dcc4474b207b9c73904219f P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:17.032433 24966 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:17.032523 24966 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:17.032613 24966 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:17.032827 24966 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:17.032828 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49c3c5c20dcc4474b207b9c73904219f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.032917 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 49c3c5c20dcc4474b207b9c73904219f (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:17.033020 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.033113 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-81b3bd116ad54a9fa10d2d0c829bb467 received from {username='slave'} at 127.14.19.134:54949
I20250110 03:07:17.033154 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-81b3bd116ad54a9fa10d2d0c829bb467 on tablet 81b3bd116ad54a9fa10d2d0c829bb467 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:17.033273 21190 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 11 to 12, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.033816 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.034682 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c3c5c20dcc4474b207b9c73904219f with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:17.034811 25129 tablet_replica.cc:331] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:17.034871 25129 raft_consensus.cc:2238] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.034924 25129 raft_consensus.cc:2267] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.035266 25129 ts_tablet_manager.cc:1905] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.036700 25129 ts_tablet_manager.cc:1918] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:17.036746 25129 log.cc:1198] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:17.037129 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 49c3c5c20dcc4474b207b9c73904219f (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:17.041250 23363 raft_consensus.cc:1212] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20250110 03:07:17.045110 23101 raft_consensus.cc:1212] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:17.048933 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54272
I20250110 03:07:17.049015 21467 raft_consensus.cc:604] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.050978 27109 ts_tablet_manager.cc:927] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:17.051133 27109 tablet_copy_client.cc:323] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:17.051417 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet abfade13c24b4cebaf1ec204947154c0 from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:49387)
I20250110 03:07:17.051468 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet abfade13c24b4cebaf1ec204947154c0 from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:49387: session id = 9da9d601b7d841b5a8f2c2c53f62a951-abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:17.052009 21488 tablet_copy_source_session.cc:215] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:17.052526 27109 ts_tablet_manager.cc:1905] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:17.054149 27109 ts_tablet_manager.cc:1918] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250110 03:07:17.054253 27109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abfade13c24b4cebaf1ec204947154c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:17.055166 27109 tablet_copy_client.cc:806] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:17.055341 27109 tablet_copy_client.cc:670] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:17.055979 27109 tablet_copy_client.cc:538] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:17.056708 27109 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:17.061796 27109 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:17.062004 27109 tablet_bootstrap.cc:492] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:17.062101 27109 ts_tablet_manager.cc:1397] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250110 03:07:17.062352 27109 raft_consensus.cc:357] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.062431 27109 raft_consensus.cc:738] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:17.062466 27109 consensus_queue.cc:260] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.062608 27109 ts_tablet_manager.cc:1428] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:17.062804 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-abfade13c24b4cebaf1ec204947154c0 received from {username='slave'} at 127.14.19.132:49387
I20250110 03:07:17.062855 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-abfade13c24b4cebaf1ec204947154c0 on tablet abfade13c24b4cebaf1ec204947154c0 with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:17.068661 21182 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet 49c3c5c20dcc4474b207b9c73904219f with cas_config_opid_index 8: aborting the task: latest config opid_index 12; task opid_index 8
I20250110 03:07:17.077433 22968 raft_consensus.cc:1212] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250110 03:07:17.088255 22968 raft_consensus.cc:1212] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:17.106420 21729 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.107012 22968 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.107050 23363 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.107182 23101 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20250110 03:07:17.107197 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.107350 21599 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.107407 25691 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.107568 27283 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.107677 25692 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.108007 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.108811 26797 raft_consensus.cc:1022] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.108870 25692 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.108989 23363 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.109059 22968 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.109933 25692 raft_consensus.cc:1059] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:17.110088 25692 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.110193 21195 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 8 to 9, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.110652 23363 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.110839 27287 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.112289 23101 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.112396 22968 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250110 03:07:17.112646 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.112728 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.112807 21599 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.113022 27283 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.113159 23100 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.113394 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.113452 21599 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.114648 25691 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.115635 25692 raft_consensus.cc:1059] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:17.115587 23363 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.115813 21196 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.115919 21599 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.116284 23100 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.117972 25692 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.118443 23363 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.118607 22968 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.118709 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.118803 21599 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.118813 23100 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.118999 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20250110 03:07:17.119074 21665 consensus_peers.cc:487] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.119103 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.119210 26797 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.119212 25691 raft_consensus.cc:1022] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.119561 22968 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.119846 26797 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.119961 23100 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.120934 22967 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.121163 21195 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.121831 23363 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.122200 21599 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.123157 21730 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.123580 23100 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.123625 22967 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.123762 25692 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.123800 23363 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.123832 27286 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.123980 27287 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.124200 21599 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.124794 27287 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.124900 27287 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.125273 23363 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.125944 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 4e11143dad6b402fb0d16c48285646c9 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.125993 21599 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.126093 23100 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.127100 21190 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 11 to 12, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.127530 22968 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.128381 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11143dad6b402fb0d16c48285646c9 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:17.128476 21730 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.128770 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 4e11143dad6b402fb0d16c48285646c9 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:17.129088 21599 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.129127 23100 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.129189 23101 raft_consensus.cc:1212] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:17.129281 25692 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.129639 25692 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.129719 22968 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.130069 27286 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.130352 25692 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.131315 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 4e11143dad6b402fb0d16c48285646c9 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.131342 23101 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.132237 22968 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.132274 21196 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.132540 21599 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.133916 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11143dad6b402fb0d16c48285646c9 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:17.134044 27068 tablet_replica.cc:331] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:17.134121 27068 raft_consensus.cc:2238] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.134204 27068 raft_consensus.cc:2267] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.134181 21730 consensus_queue.cc:237] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.134658 22968 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.134654 27068 ts_tablet_manager.cc:1905] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.134750 23101 raft_consensus.cc:1270] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.134829 27286 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.135000 27286 consensus_queue.cc:1035] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.135553 25692 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.135859 27068 ts_tablet_manager.cc:1918] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250110 03:07:17.135900 27068 log.cc:1198] T 4e11143dad6b402fb0d16c48285646c9 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:17.136219 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 4e11143dad6b402fb0d16c48285646c9 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:17.136600 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 4e11143dad6b402fb0d16c48285646c9 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.136662 21195 catalog_manager.cc:5526] T 4e11143dad6b402fb0d16c48285646c9 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.136750 23101 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.136871 22968 raft_consensus.cc:2949] T 4e11143dad6b402fb0d16c48285646c9 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.138159 27283 raft_consensus.cc:1059] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:17.138347 27283 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:17.139014 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.139017 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e11143dad6b402fb0d16c48285646c9 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:17.139055 21335 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.139159 25793 tablet_replica.cc:331] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:17.139228 25793 raft_consensus.cc:2238] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.139209 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.139354 23101 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.139355 25793 raft_consensus.cc:2267] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.139336 21599 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.139422 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.139525 23363 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.139518 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.139926 25793 ts_tablet_manager.cc:1905] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.140110 27287 raft_consensus.cc:1022] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.141237 25793 ts_tablet_manager.cc:1918] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:17.141274 25793 log.cc:1198] T 4e11143dad6b402fb0d16c48285646c9 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:17.141607 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 4e11143dad6b402fb0d16c48285646c9 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:17.141723 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.141841 25693 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.142737 27287 raft_consensus.cc:1022] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.142866 27286 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.143025 23363 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.143358 21335 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.144253 21599 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.144233 23101 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.144289 23229 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.145221 27283 raft_consensus.cc:1059] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:17.145392 21196 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.145428 27283 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.145998 23363 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.146171 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.146193 21599 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.146278 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.146667 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.146785 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.146831 23101 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.147235 21335 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.147435 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.147564 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.147676 27293 raft_consensus.cc:1022] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.148422 27287 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.148720 21599 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.149145 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54448
I20250110 03:07:17.149227 21467 raft_consensus.cc:604] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.149526 23229 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.149650 23101 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.149717 21195 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.149762 23363 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.151264 21730 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.151675 27287 raft_consensus.cc:1022] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.151858 23101 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.151973 23363 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.151978 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.152303 21599 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.152307 27283 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.152412 27283 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.152477 27283 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.152755 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.153268 27293 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.153465 23101 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.154070 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ea70c2cc77384698aedb06bee2b639ce with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.154152 21599 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.154814 21190 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 8 to 9, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.155328 27287 raft_consensus.cc:1059] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:17.155449 27287 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.155568 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet ea70c2cc77384698aedb06bee2b639ce with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:17.155704 25725 tablet_replica.cc:331] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:17.155755 25725 raft_consensus.cc:2238] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.155817 25725 pending_rounds.cc:62] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Trying to abort 1 pending ops.
I20250110 03:07:17.155861 25725 pending_rounds.cc:69] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7112615678547292160 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ea70c2cc77384698aedb06bee2b639ce" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } } }
I20250110 03:07:17.155982 25725 raft_consensus.cc:2911] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250110 03:07:17.156008 25725 raft_consensus.cc:2883] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:17.156024 25725 raft_consensus.cc:2267] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.156023 23101 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.156116 21599 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.156172 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.156356 27293 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.156469 25725 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.156724 23230 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.156998 25692 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.157215 23230 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.157791 25725 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:17.157851 25725 log.cc:1198] T ea70c2cc77384698aedb06bee2b639ce P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:17.158215 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet ea70c2cc77384698aedb06bee2b639ce (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:17.158141 27283 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.158344 23230 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.159153 23101 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.159281 21195 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.159613 21599 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.160120 21730 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.160837 23100 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.161028 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.161288 23101 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.161747 24438 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.161980 23362 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.162153 23364 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.162407 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.162575 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.162580 24438 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
W20250110 03:07:17.162634 24438 raft_consensus.cc:1062] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Unable to promote non-voter 531cf07bcce6435c9e18185c60b2118d: Illegal state: Leader has not yet committed an operation in its own term
I20250110 03:07:17.162757 23364 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.162837 21335 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.162820 27287 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.162992 25754 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.163242 27287 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.163599 23364 raft_consensus.cc:867] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
W20250110 03:07:17.163652 23364 raft_consensus.cc:1574] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.163755 23364 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.164215 23229 raft_consensus.cc:3054] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Advancing to term 2
I20250110 03:07:17.164266 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ea70c2cc77384698aedb06bee2b639ce with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.164491 23230 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.164856 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.165031 24438 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.165331 23100 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.165817 24138 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:17.165951 24138 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:17.166446 21195 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.166605 23230 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:17.166721 21336 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:17.166893 23363 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:17.167110 23100 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:17.167150 23362 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.167346 24123 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.167461 24438 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.167840 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet ea70c2cc77384698aedb06bee2b639ce with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:17.167953 25793 tablet_replica.cc:331] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:17.168001 25793 raft_consensus.cc:2238] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.168087 25793 raft_consensus.cc:2267] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.168123 23364 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.168442 25793 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.169663 25793 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:17.169708 25793 log.cc:1198] T ea70c2cc77384698aedb06bee2b639ce P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:17.167855 24438 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.002s
I20250110 03:07:17.170075 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet ea70c2cc77384698aedb06bee2b639ce (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:17.170181 24123 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.170399 23229 raft_consensus.cc:1212] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Deduplicated request from leader. Original: 1.11->[2.12-2.13] Dedup: 2.12->[2.13-2.13]
I20250110 03:07:17.170734 24123 raft_consensus.cc:1022] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: attempt to promote peer b2e7bce763e24b63bc760b0b82b53454: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.170956 21183 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet 4e11143dad6b402fb0d16c48285646c9 with cas_config_opid_index 10: aborting the task: latest config opid_index 14; task opid_index 10
I20250110 03:07:17.170951 24438 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.171424 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.172045 21336 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.172274 23100 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.172935 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Committing config change with OpId 2.13: config changed from index 11 to 13, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.173549 21188 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 11 to 13, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.173977 25753 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:17.174134 25753 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.174639 23362 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:17.174697 23100 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:17.174714 21336 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:17.174852 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:17.175036 24123 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.175120 25754 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.175215 24437 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.175266 24138 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.175940 24123 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.176079 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.176241 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.177323 23100 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.177486 21190 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.177541 21336 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.179101 21599 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.179512 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:17.179524 23100 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:17.179736 24138 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.179785 23364 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:17.179872 24138 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.180172 24437 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.180697 24138 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.180861 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.181528 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.181875 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.181928 23100 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.182274 21194 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 14 to 15, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.183441 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:17.183486 21599 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.183539 25725 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:17.183627 25725 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.183739 25725 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.183868 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:17.183977 25725 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.184013 24138 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:17.184820 24437 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.185262 25725 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.14
I20250110 03:07:17.185312 25725 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:17.185647 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.185729 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:17.185878 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.185978 21194 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 15 to 16, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.186028 23364 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:17.186371 24138 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:17.186645 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:17.186753 25129 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:17.186800 25129 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.186839 25129 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.187139 25129 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.188277 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.188424 25129 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250110 03:07:17.188478 25129 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:17.188889 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:17.193480 21467 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.193939 21599 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.193985 23229 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.193989 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.194154 25089 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.194240 25088 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250110 03:07:17.194401 21403 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.194511 25089 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.195070 27030 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.195192 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.195470 21599 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.195531 23229 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.196166 27030 raft_consensus.cc:1059] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:17.196327 27030 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.196575 21194 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 7 to 8, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.196830 21599 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.197077 22884 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250110 03:07:17.197283 21403 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.197520 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.197817 25088 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.197891 23229 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.198051 25088 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.198755 25089 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.199083 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.199823 21599 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.199899 23229 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.200163 21195 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 8 to 9, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.200268 21183 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet ea70c2cc77384698aedb06bee2b639ce with cas_config_opid_index 7: aborting the task: latest config opid_index 11; task opid_index 7
I20250110 03:07:17.201071 21467 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.201462 23229 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250110 03:07:17.201638 21403 consensus_peers.cc:487] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.201776 25088 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.201812 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.201974 25088 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.202513 22884 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.202788 23229 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.203613 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a790eedb619d47c6a3ba7eba194b70c1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.203677 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.203770 21195 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 9 to 10, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.204375 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet a790eedb619d47c6a3ba7eba194b70c1 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:17.204466 25793 tablet_replica.cc:331] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:17.204521 25793 raft_consensus.cc:2238] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.204613 25793 raft_consensus.cc:2267] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.204869 25793 ts_tablet_manager.cc:1905] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.205902 25793 ts_tablet_manager.cc:1918] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:17.205935 25793 log.cc:1198] T a790eedb619d47c6a3ba7eba194b70c1 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:17.206215 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet a790eedb619d47c6a3ba7eba194b70c1 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:17.219890 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54486
I20250110 03:07:17.219952 21467 raft_consensus.cc:604] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.233971 21181 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 13: aborting the task: latest config opid_index 16; task opid_index 13
I20250110 03:07:17.237354 25088 raft_consensus.cc:1059] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:17.237538 25088 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:17.238217 23100 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.238242 21336 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.238255 22968 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.238452 23364 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.238523 23229 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.238617 21730 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.238747 24203 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.238826 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.238898 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.238941 22874 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.239010 22884 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.239112 24065 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.239883 24221 raft_consensus.cc:1022] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.240180 24065 raft_consensus.cc:1022] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.240337 24203 raft_consensus.cc:1022] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer e4d83809ce4746d5ad65e8462857fd9c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.240640 24203 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.240890 21336 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.241818 23364 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.242159 22968 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.242915 21188 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.243139 25088 raft_consensus.cc:1059] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:17.243178 27030 raft_consensus.cc:1059] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:17.243278 25088 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:17.243628 27030 raft_consensus.cc:1062] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Unable to promote non-voter e4d83809ce4746d5ad65e8462857fd9c: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:17.243913 21336 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.243973 23100 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.244108 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.244130 22968 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.244513 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.244858 23101 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.244930 21730 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.245003 23229 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.245224 24220 raft_consensus.cc:1022] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer e4d83809ce4746d5ad65e8462857fd9c: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.245743 21729 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.246099 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.246454 23229 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.246493 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.246647 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.247123 24221 raft_consensus.cc:1022] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.247176 24221 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.248035 23229 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.248101 22968 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
W20250110 03:07:17.248427 21182 catalog_manager.cc:5049] ChangeConfig:REMOVE_PEER RPC for tablet fcde555c3b364cf3b9b582ee3e273f3e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 8 but the committed config has opid_index of 9
I20250110 03:07:17.248849 21336 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.248852 21729 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.249145 21195 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.249743 25089 raft_consensus.cc:1059] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:17.249768 23101 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.249890 25089 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.250677 23102 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.250945 21336 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.251133 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.251266 27030 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.251259 22968 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.251540 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.251619 23230 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.251796 23362 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.251824 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.252394 22874 raft_consensus.cc:1022] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer 531cf07bcce6435c9e18185c60b2118d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.252485 27030 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.252553 22874 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.252538 23230 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.253072 23362 raft_consensus.cc:867] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
W20250110 03:07:17.253121 23362 raft_consensus.cc:1574] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.253221 23362 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.253464 21729 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.253666 24220 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.253646 22968 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.253690 21336 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.253954 21729 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.254799 21194 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.255781 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.255887 23100 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.256263 22968 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.256305 23230 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.256462 25089 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.256547 22884 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.256601 21336 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.256919 22884 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.256955 23364 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.257193 22884 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.257457 24065 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.257686 21730 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.257798 24168 raft_consensus.cc:1059] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:17.257858 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.257932 24168 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.258178 21730 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.258407 22968 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.258608 25174 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.259069 23364 raft_consensus.cc:867] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
W20250110 03:07:17.259128 23364 raft_consensus.cc:1574] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 11' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }:
New config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.259181 21599 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.259231 23364 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.259377 25176 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.259653 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fcde555c3b364cf3b9b582ee3e273f3e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.259747 21188 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.260530 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:17.260463 21336 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.260536 23230 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.260638 25129 tablet_replica.cc:331] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:17.260696 25129 raft_consensus.cc:2238] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:17.260753 25129 raft_consensus.cc:2267] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:17.260746 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.261279 23362 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.261513 23229 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.261561 23231 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.261685 26778 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.261814 24168 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.261827 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.261931 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.262205 23231 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.262472 23362 raft_consensus.cc:867] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.262789 22967 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20250110 03:07:17.262524 23362 raft_consensus.cc:1574] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 12' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.263052 23362 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.263063 21599 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.263504 21336 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.264158 21188 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.264245 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.264348 22884 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.264495 22967 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.265635 24220 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.265906 22967 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.266374 22966 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.266630 23229 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.266929 23362 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.267436 21336 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.267913 21195 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.268036 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fcde555c3b364cf3b9b582ee3e273f3e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.268632 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:17.268730 25921 tablet_replica.cc:331] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:17.268788 25921 raft_consensus.cc:2238] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.268915 25921 raft_consensus.cc:2267] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.269050 23364 raft_consensus.cc:1212] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20250110 03:07:17.269294 25921 ts_tablet_manager.cc:1905] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.269449 25129 ts_tablet_manager.cc:1905] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.270591 25129 ts_tablet_manager.cc:1918] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:17.270591 25921 ts_tablet_manager.cc:1918] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:17.270650 25129 log.cc:1198] T fcde555c3b364cf3b9b582ee3e273f3e P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:17.270648 25921 log.cc:1198] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:17.270918 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.271065 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet fcde555c3b364cf3b9b582ee3e273f3e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:17.271214 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet fcde555c3b364cf3b9b582ee3e273f3e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:17.271436 23231 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.271466 23364 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.271600 22968 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.271612 24203 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.271710 25089 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.272274 27030 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.272408 23231 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.272274 25089 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20250110 03:07:17.273594 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fcde555c3b364cf3b9b582ee3e273f3e with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.273705 21190 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.274406 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:17.274502 25725 tablet_replica.cc:331] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:17.274586 25725 raft_consensus.cc:2238] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.274732 25725 raft_consensus.cc:2267] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.275007 25725 ts_tablet_manager.cc:1905] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.275136 23364 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.276136 22968 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.276701 25725 ts_tablet_manager.cc:1918] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250110 03:07:17.276757 25725 log.cc:1198] T fcde555c3b364cf3b9b582ee3e273f3e P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:17.276813 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.277109 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet fcde555c3b364cf3b9b582ee3e273f3e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:17.277313 23231 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.277350 23364 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.277467 24203 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.277542 25089 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.277776 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54546
I20250110 03:07:17.277830 21467 raft_consensus.cc:604] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.278009 25089 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.279124 23364 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.279317 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fcde555c3b364cf3b9b582ee3e273f3e with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.279536 23231 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.280226 21188 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 13 to 14, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.281059 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:17.281157 25867 tablet_replica.cc:331] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:17.281219 25867 raft_consensus.cc:2238] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.281282 25867 raft_consensus.cc:2267] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.281625 25867 ts_tablet_manager.cc:1905] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.283005 25867 ts_tablet_manager.cc:1918] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:17.283069 25867 log.cc:1198] T fcde555c3b364cf3b9b582ee3e273f3e P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:17.283468 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet fcde555c3b364cf3b9b582ee3e273f3e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:17.301834 21336 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.302268 21730 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.302356 23100 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.302524 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.302717 23231 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.303499 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.303773 24152 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.304286 24152 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.305369 23231 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.305600 21730 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.306818 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 58e33e63f48f4b758c19774888345ee0 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.306763 21190 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.306869 23100 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.307677 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e33e63f48f4b758c19774888345ee0 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:17.307768 25867 tablet_replica.cc:331] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:17.307734 21336 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.307814 25867 raft_consensus.cc:2238] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.307876 25867 raft_consensus.cc:2267] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.308051 21730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce"
dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35808
I20250110 03:07:17.308127 21730 raft_consensus.cc:604] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.308244 25867 ts_tablet_manager.cc:1905] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.308645 23231 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.308830 26816 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.309433 26816 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.309682 25867 ts_tablet_manager.cc:1918] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:17.309741 25867 log.cc:1198] T 58e33e63f48f4b758c19774888345ee0 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/58e33e63f48f4b758c19774888345ee0
I20250110 03:07:17.310211 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 58e33e63f48f4b758c19774888345ee0 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:17.311580 21188 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.312389 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 58e33e63f48f4b758c19774888345ee0 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.312834 23231 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.312906 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e33e63f48f4b758c19774888345ee0 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:17.312988 25921 tablet_replica.cc:331] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:17.313040 25921 raft_consensus.cc:2238] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.313151 25921 raft_consensus.cc:2267] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.313441 25921 ts_tablet_manager.cc:1905] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.313982 23100 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.314247 26778 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.314674 23100 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.314874 25921 ts_tablet_manager.cc:1918] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:17.314922 25921 log.cc:1198] T 58e33e63f48f4b758c19774888345ee0 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/58e33e63f48f4b758c19774888345ee0
I20250110 03:07:17.315339 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 58e33e63f48f4b758c19774888345ee0 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:17.318686 21467 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 03:07:17.318892 27283 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.321277 23231 raft_consensus.cc:1212] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250110 03:07:17.323400 21467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d"
dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54660
I20250110 03:07:17.323468 21467 raft_consensus.cc:604] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.356334 26778 raft_consensus.cc:1059] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb to VOTER
I20250110 03:07:17.356554 26778 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:17.357336 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.357405 21467 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.357481 23364 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.357563 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.357688 25174 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.358065 23100 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.358253 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.358330 21599 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:17.358366 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.359246 25174 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.359393 25174 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.359525 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.360673 21467 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.360883 21190 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.361202 21599 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.362960 24152 raft_consensus.cc:1059] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:17.363096 24152 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.363483 23362 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.363605 21467 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.363644 26816 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.363754 26816 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.364005 23362 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.364184 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.364374 25174 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.364869 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.365119 26778 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.365192 23102 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:17.365536 26778 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:17.365656 26778 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.365785 23364 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.366039 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.366724 23102 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:17.366828 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.367136 21187 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.367677 21467 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.369486 21336 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.369946 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.369946 23364 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.370035 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.370103 26778 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.370173 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.370236 21467 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.370267 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.370429 24168 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.371273 24168 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.371567 21467 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.372506 23364 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.372478 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.374015 21187 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.374089 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.374143 24203 raft_consensus.cc:1059] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:17.374244 21730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce"
dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35856
I20250110 03:07:17.374294 21730 raft_consensus.cc:604] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.374298 24203 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.374015 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.374840 23231 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.374872 21336 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.375180 23102 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.375226 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:17.375309 25129 tablet_replica.cc:331] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:17.375362 25129 raft_consensus.cc:2238] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:17.375409 25129 pending_rounds.cc:62] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Trying to abort 1 pending ops.
I20250110 03:07:17.375434 25129 pending_rounds.cc:69] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7112615679439020032 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7dbef1612c8b4e2baa670b1cbe0e8960" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } } }
I20250110 03:07:17.375550 25129 raft_consensus.cc:2911] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20250110 03:07:17.375582 25129 raft_consensus.cc:2883] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:17.375608 25129 raft_consensus.cc:2267] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:17.375926 25089 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.376410 22968 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.376678 25089 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.376754 22884 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.376879 22884 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.377076 21336 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.377662 23364 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.377660 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.377780 21467 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.377820 24726 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.378319 24726 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.378566 25089 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.379046 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.379491 24726 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.379711 21195 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.379743 23231 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.380553 23364 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.380952 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.381111 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.381129 22968 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.381830 21467 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.381815 21195 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.382591 21467 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.382611 21336 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.382794 23231 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.382974 24203 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.383417 22968 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.383509 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:17.383607 25921 tablet_replica.cc:331] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:17.383679 25921 raft_consensus.cc:2238] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.383677 21337 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.383733 24203 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.383785 25921 raft_consensus.cc:2267] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.384034 21335 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.384106 23364 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.384258 25921 ts_tablet_manager.cc:1905] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.384271 25129 ts_tablet_manager.cc:1905] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.384301 26778 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.384941 26778 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.385104 23364 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.385649 25129 ts_tablet_manager.cc:1918] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:17.385704 25129 log.cc:1198] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:17.385854 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.385838 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.386073 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:17.386353 27030 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.386308 21188 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.386366 24152 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.386739 25921 ts_tablet_manager.cc:1918] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:17.386793 25921 log.cc:1198] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:17.386860 27030 raft_consensus.cc:988] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: : Instructing follower e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:17.386912 27030 raft_consensus.cc:1076] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Signalling peer e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:17.386865 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.387138 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:17.387178 22966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d"
dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
from {username='slave'} at 127.14.19.130:32997
I20250110 03:07:17.387259 22966 raft_consensus.cc:491] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:17.387328 22966 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.387988 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:17.388092 25197 tablet_replica.cc:331] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:17.388082 22966 raft_consensus.cc:513] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.388157 25197 raft_consensus.cc:2238] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.388250 25197 raft_consensus.cc:2267] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.388506 25197 ts_tablet_manager.cc:1905] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.388614 22966 leader_election.cc:290] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 2 election: Requested vote from peers ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469), b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887), d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:17.389634 24203 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.389770 25197 ts_tablet_manager.cc:1918] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:17.389822 25197 log.cc:1198] T 7dbef1612c8b4e2baa670b1cbe0e8960 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:17.389814 22968 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.390208 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:17.390374 21467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "ab3ed3d8c2394f78943461e22853c7ac"
I20250110 03:07:17.390446 21467 raft_consensus.cc:3049] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:17.390477 21467 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Running, Role: LEADER
I20250110 03:07:17.390516 21467 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.390654 21467 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.390893 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3b266cc7c4f341f785152932febcb79b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.390916 23231 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.391057 21190 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.391225 21467 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:17.391990 21335 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.392076 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b266cc7c4f341f785152932febcb79b with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:17.392174 25129 tablet_replica.cc:331] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:17.392238 25129 raft_consensus.cc:2238] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus shutting down.
I20250110 03:07:17.392274 25129 pending_rounds.cc:62] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Trying to abort 1 pending ops.
I20250110 03:07:17.392310 25129 pending_rounds.cc:69] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Aborting op as it isn't in flight: id { term: 1 index: 9 } timestamp: 7112615679484731392 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3b266cc7c4f341f785152932febcb79b" old_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } } new_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } } }
I20250110 03:07:17.392406 25129 raft_consensus.cc:2911] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Aborting config change with OpId 1.9: Aborted: Op aborted
I20250110 03:07:17.392438 25129 raft_consensus.cc:2883] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:17.392455 25129 raft_consensus.cc:2267] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Raft consensus is shut down!
I20250110 03:07:17.392747 25129 ts_tablet_manager.cc:1905] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.394411 25129 ts_tablet_manager.cc:1918] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:17.394596 25129 log.cc:1198] T 3b266cc7c4f341f785152932febcb79b P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/3b266cc7c4f341f785152932febcb79b
I20250110 03:07:17.394767 23231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "b2e7bce763e24b63bc760b0b82b53454"
I20250110 03:07:17.394857 23231 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.395020 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 3b266cc7c4f341f785152932febcb79b (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:17.395520 23231 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:17.395673 22906 leader_election.cc:304] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ab3ed3d8c2394f78943461e22853c7ac, b2e7bce763e24b63bc760b0b82b53454, e4d83809ce4746d5ad65e8462857fd9c; no voters:
I20250110 03:07:17.395789 24788 raft_consensus.cc:2798] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:17.395911 24788 raft_consensus.cc:695] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Becoming Leader. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Running, Role: LEADER
I20250110 03:07:17.395984 24788 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.396884 21195 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: term changed from 1 to 2, leader changed from ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) to e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.397030 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0760e9b8e3c642f09c3287ca586a2a1d" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
I20250110 03:07:17.397099 23364 raft_consensus.cc:3054] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.397819 23364 raft_consensus.cc:2463] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:17.409832 21730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce"
dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35884
I20250110 03:07:17.409916 21730 raft_consensus.cc:604] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.428999 22968 raft_consensus.cc:1212] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:17.430723 21335 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.431154 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.431206 23231 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20250110 03:07:17.431264 21274 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.431414 25176 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.431500 26778 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.432066 25176 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.432229 23231 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.432255 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.433194 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 62aacc8deebd43debd76a2c10815da2f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.433336 21190 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 9 to 10, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.434041 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 62aacc8deebd43debd76a2c10815da2f with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:17.434159 25197 tablet_replica.cc:331] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:17.434250 25197 raft_consensus.cc:2238] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.434340 25197 raft_consensus.cc:2267] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.434597 25197 ts_tablet_manager.cc:1905] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.435959 25197 ts_tablet_manager.cc:1918] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:07:17.436005 25197 log.cc:1198] T 62aacc8deebd43debd76a2c10815da2f P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:17.436288 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 62aacc8deebd43debd76a2c10815da2f (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:17.470813 22884 raft_consensus.cc:1059] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:17.471000 22884 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.471736 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.471745 23362 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.471791 21335 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.471997 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.472046 23102 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.472126 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.472420 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.472733 24221 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.473165 23231 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.473518 24221 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.473634 24220 raft_consensus.cc:1022] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: attempt to promote peer d0be0af85dee41dd8d01ef4b2af7726f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:17.473958 24221 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.474105 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.474850 21335 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.475216 21188 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.475297 23362 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.475658 27030 raft_consensus.cc:1059] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:17.475777 27030 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.476013 23102 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.476743 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.476935 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.477031 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.477319 23229 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.477355 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.477439 23231 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.477797 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.478245 23362 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.479106 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.479264 25088 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.479548 23229 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.479836 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.480518 21335 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.480564 24203 raft_consensus.cc:1059] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:17.480716 24203 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.480883 21187 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.481128 23229 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.481340 23364 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.481413 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.481412 23362 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.481688 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.481642 23100 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.481727 21335 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.481905 25088 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.481962 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.482299 21335 raft_consensus.cc:867] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
W20250110 03:07:17.482344 21335 raft_consensus.cc:1574] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 12' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }:
New config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.482414 21335 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.482908 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.483078 25088 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.483516 25088 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.483825 25089 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.484689 23362 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.484714 23229 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.484753 21188 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 11 to 12, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.484859 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.485922 23100 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.486083 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.486670 23229 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.486862 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.486991 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.487180 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.487314 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.487388 23362 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.487588 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.487689 25088 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.487951 25088 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.488412 23100 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.488842 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.489006 23229 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.489322 21188 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.489598 23362 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.490228 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet b5064ab85103488f873fffef55fd78fd with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:17.490350 25725 tablet_replica.cc:331] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:17.490414 25725 raft_consensus.cc:2238] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 03:07:17.490437 25089 consensus_queue.cc:1159] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b2e7bce763e24b63bc760b0b82b53454" responder_term: 1 status { last_received { term: 1 index: 13 } last_committed_idx: 13 last_received_current_leader { term: 1 index: 13 } }
I20250110 03:07:17.490522 25725 pending_rounds.cc:62] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Trying to abort 1 pending ops.
I20250110 03:07:17.490497 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.490577 25725 pending_rounds.cc:69] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Aborting op as it isn't in flight: id { term: 1 index: 12 } timestamp: 7112615679920840704 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b5064ab85103488f873fffef55fd78fd" old_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } } new_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } } }
I20250110 03:07:17.490710 25725 raft_consensus.cc:2911] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Aborting config change with OpId 1.12: Aborted: Op aborted
I20250110 03:07:17.490742 25725 raft_consensus.cc:2883] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250110 03:07:17.490768 25725 raft_consensus.cc:2267] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.490988 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.491034 25725 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.491154 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.492307 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.492389 25725 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250110 03:07:17.492437 25725 log.cc:1198] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/b5064ab85103488f873fffef55fd78fd
I20250110 03:07:17.492789 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet b5064ab85103488f873fffef55fd78fd (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:17.493009 22966 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.493235 25089 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.493662 23364 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.493906 24221 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.493907 24220 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.494200 23362 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.494885 22966 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.495409 21196 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.496126 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet b5064ab85103488f873fffef55fd78fd with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:17.496234 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.496465 23100 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.496629 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.497004 22966 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.497176 23102 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.497246 27742 tablet_replica.cc:331] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:17.497315 27742 raft_consensus.cc:2238] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.497442 24221 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.497522 24220 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.497638 27742 raft_consensus.cc:2267] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.497947 27742 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.498113 24220 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.498273 23102 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.499042 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.499470 27742 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:17.499568 27742 log.cc:1198] T b5064ab85103488f873fffef55fd78fd P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/b5064ab85103488f873fffef55fd78fd
I20250110 03:07:17.499908 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet b5064ab85103488f873fffef55fd78fd (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:17.502635 21196 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 14 to 15, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.502796 22966 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:17.503652 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet b5064ab85103488f873fffef55fd78fd with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:17.503741 27068 tablet_replica.cc:331] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:17.503790 27068 raft_consensus.cc:2238] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.503821 27068 raft_consensus.cc:2267] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.504065 27068 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.505259 27068 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250110 03:07:17.505306 27068 log.cc:1198] T b5064ab85103488f873fffef55fd78fd P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/b5064ab85103488f873fffef55fd78fd
I20250110 03:07:17.505631 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet b5064ab85103488f873fffef55fd78fd (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:17.522766 23362 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 03:07:17.522984 25692 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:17.524026 21598 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 03:07:17.524228 26797 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250110 03:07:17.549163 21182 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 11: aborting the task: latest config opid_index 15; task opid_index 11
I20250110 03:07:17.578974 21730 raft_consensus.cc:1212] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:17.588635 23102 raft_consensus.cc:1212] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:17.611397 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:17.611960 21710 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:17.617179 23344 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:17.617590 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:17.619982 22951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:17.622383 21318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:17.624742 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:17.628455 21580 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20250110 03:07:17.648113 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.648528 23362 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.648573 23229 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20250110 03:07:17.648626 21405 consensus_peers.cc:487] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.648757 25089 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.648849 24221 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.649461 27030 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.649597 23229 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.649725 23362 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.650889 21190 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.652671 21730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce"
dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35964
I20250110 03:07:17.652781 21730 raft_consensus.cc:604] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.656054 21467 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:17.656585 22966 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:17.656581 21404 consensus_peers.cc:487] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.656597 23102 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.656762 25089 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.656853 24221 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.657631 27030 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.657792 22966 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.658843 23102 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.658979 21194 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 12 to 13, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.660777 21730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce"
dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35988
I20250110 03:07:17.660837 21730 raft_consensus.cc:604] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.662858 24152 raft_consensus.cc:1059] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:17.663004 24152 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.663447 23229 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.663502 22966 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.663571 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:17.663769 24168 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.664086 24168 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.664224 25176 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:17.664840 24168 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.665402 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.666175 22966 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.666599 21189 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.667383 23229 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.696612 21318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:17.696712 21710 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:17.699123 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:17.702868 22951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:17.703117 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:17.703547 21580 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20250110 03:07:17.710500 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:17.712443 28003 ts_tablet_manager.cc:927] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:17.712630 28003 tablet_copy_client.cc:323] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:17.712862 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet a790eedb619d47c6a3ba7eba194b70c1 from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:33889)
I20250110 03:07:17.712909 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet a790eedb619d47c6a3ba7eba194b70c1 from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:33889: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:17.713393 21488 tablet_copy_source_session.cc:215] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:17.713757 28003 ts_tablet_manager.cc:1905] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:17.715651 28003 ts_tablet_manager.cc:1918] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250110 03:07:17.715754 28003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a790eedb619d47c6a3ba7eba194b70c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:17.716382 28003 tablet_copy_client.cc:806] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:17.716497 28003 tablet_copy_client.cc:670] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:17.719396 28003 tablet_copy_client.cc:538] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:17.719534 23344 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:17.720166 28003 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:17.723620 28003 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:17.723816 28003 tablet_bootstrap.cc:492] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:17.723901 28003 ts_tablet_manager.cc:1397] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250110 03:07:17.724006 28003 raft_consensus.cc:357] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.724067 28003 raft_consensus.cc:738] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:17.724090 28003 consensus_queue.cc:260] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.724203 28003 ts_tablet_manager.cc:1428] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:07:17.724468 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-a790eedb619d47c6a3ba7eba194b70c1 received from {username='slave'} at 127.14.19.129:33889
I20250110 03:07:17.724519 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-a790eedb619d47c6a3ba7eba194b70c1 on tablet a790eedb619d47c6a3ba7eba194b70c1 with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:17.731678 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:17.732178 23362 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:17.732270 21534 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.732270 23102 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.732471 24437 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.732564 24165 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.733335 24138 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.733474 23362 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.734616 27286 raft_consensus.cc:988] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: : Instructing follower d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:17.734678 27286 raft_consensus.cc:1076] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Signalling peer d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:17.734732 21189 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.734858 23362 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
from {username='slave'} at 127.14.19.132:34013
I20250110 03:07:17.734913 23362 raft_consensus.cc:491] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:17.734932 23362 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.735067 23102 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:17.735590 23362 raft_consensus.cc:513] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.735766 23362 leader_election.cc:290] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887), 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351)
I20250110 03:07:17.736549 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:17.736660 21730 raft_consensus.cc:3049] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:17.736689 21730 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Running, Role: LEADER
I20250110 03:07:17.736732 21730 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.736832 21730 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.737465 21730 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:17.739498 23229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "b2e7bce763e24b63bc760b0b82b53454"
I20250110 03:07:17.739575 23229 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.740171 23229 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:17.740271 23362 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:17.740305 23299 leader_election.cc:304] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, b2e7bce763e24b63bc760b0b82b53454, d0be0af85dee41dd8d01ef4b2af7726f; no voters:
I20250110 03:07:17.740427 26732 raft_consensus.cc:2798] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:17.740509 26732 raft_consensus.cc:695] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Becoming Leader. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Running, Role: LEADER
I20250110 03:07:17.740558 26732 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:17.740911 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.741155 21194 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: term changed from 1 to 2, leader changed from 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) to d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136). New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.741335 21598 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.741362 23102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea70c2cc77384698aedb06bee2b639ce" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "531cf07bcce6435c9e18185c60b2118d"
I20250110 03:07:17.741444 23102 raft_consensus.cc:3054] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:17.741956 23102 raft_consensus.cc:2463] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:17.741112 28022 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.742816 28024 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.743639 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.744127 21598 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.743556 28023 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.745887 21189 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 11 to 12, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.750046 21335 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
W20250110 03:07:17.750129 23298 consensus_peers.cc:487] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.750515 23362 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.750646 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.750710 25176 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250110 03:07:17.750797 21274 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.750813 26778 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.751459 26816 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.751572 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.751572 23364 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.752707 21196 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 11 to 12, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.756752 21598 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:17.757130 23362 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250110 03:07:17.757182 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
W20250110 03:07:17.757280 21535 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): Couldn't send request to peer ab3ed3d8c2394f78943461e22853c7ac. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.757380 24165 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:17.757431 24138 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:17.757993 24138 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:17.758173 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:17.758773 23362 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:17.759284 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 16 to 17, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.762707 21335 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.763128 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.763124 23229 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.763258 22966 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.763314 24168 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.763407 25176 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.763480 26778 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.764133 25176 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.764245 22966 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.765115 23229 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.765117 21196 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 10 to 11. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.765164 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.767014 21335 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.767426 22966 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.767541 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.767732 26778 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.767836 26816 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.768242 26778 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.768445 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.769192 22966 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.769449 21189 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 11 to 12, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.770354 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.770641 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 250f547c54a84a35b2850d50b897810e with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:17.770776 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:17.770843 21335 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:17.771016 27742 tablet_replica.cc:331] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:17.771082 27742 raft_consensus.cc:2238] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.771165 27742 raft_consensus.cc:2267] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.771245 22966 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.771412 24221 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.771435 21335 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:17.771457 21272 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 250f547c54a84a35b2850d50b897810e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.771533 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:17.771550 21403 consensus_peers.cc:487] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.771554 22966 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.771736 25176 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.771837 27742 ts_tablet_manager.cc:1905] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.771842 25176 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.772357 26778 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.773038 24203 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.773180 27742 ts_tablet_manager.cc:1918] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:17.773237 27742 log.cc:1198] T 250f547c54a84a35b2850d50b897810e P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/250f547c54a84a35b2850d50b897810e
I20250110 03:07:17.773195 24203 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.773312 21181 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:07:17.773653 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 250f547c54a84a35b2850d50b897810e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:17.773680 21196 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.773780 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.774776 21191 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.774838 22968 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.775193 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.776129 21335 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:17.780256 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:17.780725 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.780787 21730 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20250110 03:07:17.780823 21535 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.780912 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.781108 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.781745 24165 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.781926 21730 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.783203 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:17.783463 21192 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 14 to 15, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.790529 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.790807 24788 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.808907 21335 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:17.809309 22968 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.809356 23229 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:17.809558 26816 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.809648 24168 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:17.810424 26816 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.810668 22968 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.810899 23229 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:17.812270 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5a1a954889a9452f8a8ee55aa0a6f26e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.812281 21196 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 8 to 9, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.813315 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1a954889a9452f8a8ee55aa0a6f26e with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:17.813642 28115 tablet_replica.cc:331] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:17.813707 28115 raft_consensus.cc:2238] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.813804 28115 raft_consensus.cc:2267] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.814102 28115 ts_tablet_manager.cc:1905] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.815618 28115 ts_tablet_manager.cc:1918] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:17.815670 28115 log.cc:1198] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:17.816090 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 5a1a954889a9452f8a8ee55aa0a6f26e (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:17.822063 21580 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20250110 03:07:17.823022 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:17.828420 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:17.828465 28132 ts_tablet_manager.cc:927] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:17.828620 28132 tablet_copy_client.cc:323] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:17.829013 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 62aacc8deebd43debd76a2c10815da2f from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:42117)
I20250110 03:07:17.829080 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 62aacc8deebd43debd76a2c10815da2f from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:42117: session id = 9da9d601b7d841b5a8f2c2c53f62a951-62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:17.829291 23344 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:17.829664 21358 tablet_copy_source_session.cc:215] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:17.829733 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:17.830050 28132 ts_tablet_manager.cc:1905] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:17.830520 21318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:17.835285 22951 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:17.838620 28132 ts_tablet_manager.cc:1918] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250110 03:07:17.838829 28132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:17.839689 28132 tablet_copy_client.cc:806] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:17.839949 28132 tablet_copy_client.cc:670] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:17.844372 28132 tablet_copy_client.cc:538] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:17.844995 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:17.845167 28132 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:17.849242 21335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "250f547c54a84a35b2850d50b897810e"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48722
I20250110 03:07:17.849345 21335 raft_consensus.cc:604] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:17.849766 28132 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:17.849980 28132 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:17.850069 28132 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20250110 03:07:17.850171 28132 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.850237 28132 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:17.850268 28132 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.850368 28132 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:17.850862 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-62aacc8deebd43debd76a2c10815da2f received from {username='slave'} at 127.14.19.132:42117
I20250110 03:07:17.850914 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-62aacc8deebd43debd76a2c10815da2f on tablet 62aacc8deebd43debd76a2c10815da2f with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:17.860229 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:17.860757 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.860812 23362 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20250110 03:07:17.860877 21536 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.860986 24138 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.861073 24165 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.861863 24165 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.862731 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.862922 23362 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:17.863924 21189 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 14 to 15, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.869232 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.869674 23362 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.869776 23100 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.869776 23229 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:17.869879 24165 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.870153 24138 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.870278 24165 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:17.871075 24437 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.871197 23229 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.871243 23100 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.872074 23362 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.873145 21188 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 13 to 14. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.874321 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:17.874687 23362 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.874753 23100 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:17.874912 24165 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.875041 24138 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:17.875416 24437 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.875517 23100 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.876286 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet f1e7baec841a4b7e9cb933d786ec2552 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.876354 21195 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.876581 23362 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:17.877038 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e7baec841a4b7e9cb933d786ec2552 with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:17.877161 27742 tablet_replica.cc:331] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:17.877233 27742 raft_consensus.cc:2238] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.877318 27742 raft_consensus.cc:2267] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.877645 27742 ts_tablet_manager.cc:1905] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.877941 21598 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
W20250110 03:07:17.878480 21534 consensus_peers.cc:487] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1e7baec841a4b7e9cb933d786ec2552. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:17.878525 23362 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:17.878664 23100 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:17.878733 24437 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:17.879171 28177 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:17.879225 27742 ts_tablet_manager.cc:1918] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250110 03:07:17.879268 27742 log.cc:1198] T f1e7baec841a4b7e9cb933d786ec2552 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:17.879323 24138 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.879626 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet f1e7baec841a4b7e9cb933d786ec2552 (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:17.880019 21181 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f1e7baec841a4b7e9cb933d786ec2552 with cas_config_opid_index 15: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:07:17.880159 24165 raft_consensus.cc:1059] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:17.880251 21194 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 15 to 16, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:17.880308 24165 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.880494 23362 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.880897 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.881224 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.881256 23364 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.881578 24138 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.882086 22968 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.882131 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:17.882349 24437 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.882352 23100 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:17.882565 28184 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:17.883059 24437 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.883883 23364 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.884339 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.884490 24138 raft_consensus.cc:1059] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:17.884506 22968 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.885600 21194 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.885912 24138 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.886526 23102 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.886718 22968 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.886731 23364 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.886917 24437 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.887755 28185 consensus_queue.cc:1037] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.887850 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.887836 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:17.888105 28185 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.888221 28185 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.889098 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.889123 24165 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.889278 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.890182 23364 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.890385 21195 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.890734 22968 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.892040 21598 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:17.892551 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.892791 28184 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.893112 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.893113 22968 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:17.893285 28184 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.893388 28185 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:17.893980 28184 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.894737 22968 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.895030 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:17.895239 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.896181 21195 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:17.896659 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:17.898089 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:17.898226 27068 tablet_replica.cc:331] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:17.898296 27068 raft_consensus.cc:2238] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:17.898375 27068 raft_consensus.cc:2267] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:17.898667 27068 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:17.899726 27068 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250110 03:07:17.899777 27068 log.cc:1198] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:17.900094 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 81b3bd116ad54a9fa10d2d0c829bb467 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:17.963583 21467 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.963835 24788 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.965153 23229 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:17.965430 28291 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:17.985976 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44654
I20250110 03:07:17.986052 21598 raft_consensus.cc:604] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:18.066965 21335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "250f547c54a84a35b2850d50b897810e"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48780
I20250110 03:07:18.067037 21335 raft_consensus.cc:604] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:18.077101 21335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "250f547c54a84a35b2850d50b897810e"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48812
I20250110 03:07:18.077169 21335 raft_consensus.cc:604] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:18.094334 21730 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:18.094780 23364 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:18.094780 21598 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:18.095001 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:18.095113 26797 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:18.095834 27283 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:18.096004 21598 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:18.096725 23364 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:18.096935 21194 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 7 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:18.097699 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 80152844286545c68c7656ef9bbe8604 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.097802 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 80152844286545c68c7656ef9bbe8604 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:18.097960 28391 tablet_replica.cc:331] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:18.098037 28391 raft_consensus.cc:2238] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.098099 28391 raft_consensus.cc:2267] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.098326 28391 ts_tablet_manager.cc:1905] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.099661 28391 ts_tablet_manager.cc:1918] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20250110 03:07:18.099709 28391 log.cc:1198] T 80152844286545c68c7656ef9bbe8604 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/80152844286545c68c7656ef9bbe8604
I20250110 03:07:18.100046 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 80152844286545c68c7656ef9bbe8604 (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:18.118279 28392 ts_tablet_manager.cc:927] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:18.118464 28392 tablet_copy_client.cc:323] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.136:42591
I20250110 03:07:18.118772 23370 tablet_copy_service.cc:140] P d0be0af85dee41dd8d01ef4b2af7726f: Received BeginTabletCopySession request for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:35405)
I20250110 03:07:18.118834 23370 tablet_copy_service.cc:161] P d0be0af85dee41dd8d01ef4b2af7726f: Beginning new tablet copy session on tablet 44bb3816ff5a4d44a249b8bbabdc7b04 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:35405: session id = e4d83809ce4746d5ad65e8462857fd9c-44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:18.119246 23370 tablet_copy_source_session.cc:215] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.119594 28392 ts_tablet_manager.cc:1905] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.120868 28392 ts_tablet_manager.cc:1918] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250110 03:07:18.120965 28392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44bb3816ff5a4d44a249b8bbabdc7b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.121611 24220 raft_consensus.cc:1059] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:18.121637 28392 tablet_copy_client.cc:806] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.121743 28392 tablet_copy_client.cc:670] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.121794 24220 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:18.122298 23364 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:18.122391 23100 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:18.122433 21730 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:18.122498 24221 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:18.122618 24220 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:18.122738 24220 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:18.122756 28392 tablet_copy_client.cc:538] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.123258 28392 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:18.123436 27030 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.123553 23364 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.124612 21730 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.124776 21189 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:18.126377 23100 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.127130 21467 consensus_queue.cc:237] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:18.127490 21730 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:18.127671 25088 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.127707 23100 raft_consensus.cc:1270] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:18.127935 25088 consensus_queue.cc:1035] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.128233 24220 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.128355 28392 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.128566 28392 tablet_bootstrap.cc:492] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:18.128649 28392 ts_tablet_manager.cc:1397] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250110 03:07:18.128589 23100 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.128746 28392 raft_consensus.cc:357] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:18.128788 28392 raft_consensus.cc:738] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:18.128826 28392 consensus_queue.cc:260] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:18.128906 28392 ts_tablet_manager.cc:1428] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.129117 23370 tablet_copy_service.cc:342] P d0be0af85dee41dd8d01ef4b2af7726f: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-44bb3816ff5a4d44a249b8bbabdc7b04 received from {username='slave'} at 127.14.19.133:35405
I20250110 03:07:18.129159 23370 tablet_copy_service.cc:434] P d0be0af85dee41dd8d01ef4b2af7726f: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-44bb3816ff5a4d44a249b8bbabdc7b04 on tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:18.129364 21730 raft_consensus.cc:2949] T abfade13c24b4cebaf1ec204947154c0 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.129688 21189 catalog_manager.cc:5526] T abfade13c24b4cebaf1ec204947154c0 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:18.130386 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet abfade13c24b4cebaf1ec204947154c0 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.130426 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet abfade13c24b4cebaf1ec204947154c0 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:18.130510 27068 tablet_replica.cc:331] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:18.130584 27068 raft_consensus.cc:2238] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.130656 27068 raft_consensus.cc:2267] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.130986 27068 ts_tablet_manager.cc:1905] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.131954 27068 ts_tablet_manager.cc:1918] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:18.131996 27068 log.cc:1198] T abfade13c24b4cebaf1ec204947154c0 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:18.132314 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet abfade13c24b4cebaf1ec204947154c0 (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:18.139803 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:18.139995 28003 ts_tablet_manager.cc:927] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:18.140017 28024 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.140161 28003 tablet_copy_client.cc:323] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:18.140444 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:51055)
I20250110 03:07:18.140506 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:51055: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:18.140977 21620 tablet_copy_source_session.cc:215] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.141248 28003 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.142457 28003 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20250110 03:07:18.142537 28003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.143066 28003 tablet_copy_client.cc:806] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.143165 28003 tablet_copy_client.cc:670] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.143680 28003 tablet_copy_client.cc:538] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.144145 28003 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:18.147156 28003 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.147333 28003 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:18.147419 28003 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250110 03:07:18.147521 28003 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:18.147583 28003 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:18.147614 28003 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:18.147708 28003 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.147898 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-41fa30056c0a4b30ab8fcaa14645e7a4 received from {username='slave'} at 127.14.19.129:51055
I20250110 03:07:18.147956 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-41fa30056c0a4b30ab8fcaa14645e7a4 on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:18.158936 28405 ts_tablet_manager.cc:927] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:18.159104 28405 tablet_copy_client.cc:323] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:18.159297 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet d52b50e3c87b47bead3a25f7255d1c42 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:43997)
I20250110 03:07:18.159363 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet d52b50e3c87b47bead3a25f7255d1c42 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:43997: session id = b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:18.159732 21488 tablet_copy_source_session.cc:215] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.160020 28405 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.161271 21335 raft_consensus.cc:1212] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:18.161440 28405 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20250110 03:07:18.161548 28405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52b50e3c87b47bead3a25f7255d1c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.162277 28405 tablet_copy_client.cc:806] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.162429 28405 tablet_copy_client.cc:670] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.163005 28405 tablet_copy_client.cc:538] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.163486 28405 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:18.167680 28405 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.167881 28405 tablet_bootstrap.cc:492] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:18.168010 28405 ts_tablet_manager.cc:1397] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250110 03:07:18.168110 28405 raft_consensus.cc:357] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.168181 28405 raft_consensus.cc:738] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:18.168228 28405 consensus_queue.cc:260] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.168326 28405 ts_tablet_manager.cc:1428] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.168515 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42 received from {username='slave'} at 127.14.19.135:43997
I20250110 03:07:18.168574 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-d52b50e3c87b47bead3a25f7255d1c42 on tablet d52b50e3c87b47bead3a25f7255d1c42 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:18.193637 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44700
I20250110 03:07:18.193727 21598 raft_consensus.cc:604] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:18.206703 28132 ts_tablet_manager.cc:927] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:18.206846 28132 tablet_copy_client.cc:323] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:18.207058 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet fcde555c3b364cf3b9b582ee3e273f3e from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:49387)
I20250110 03:07:18.207120 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet fcde555c3b364cf3b9b582ee3e273f3e from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:49387: session id = 9da9d601b7d841b5a8f2c2c53f62a951-fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:18.207540 28490 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:18.207636 21488 tablet_copy_source_session.cc:215] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.207685 28490 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:18.207875 28132 ts_tablet_manager.cc:1905] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.207895 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer ab3ed3d8c2394f78943461e22853c7ac ({username='slave'} at 127.14.19.130:43035)
I20250110 03:07:18.207947 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer ab3ed3d8c2394f78943461e22853c7ac at {username='slave'} at 127.14.19.130:43035: session id = ab3ed3d8c2394f78943461e22853c7ac-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:18.208463 21620 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.208832 28490 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.209764 28132 ts_tablet_manager.cc:1918] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20250110 03:07:18.209863 28132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcde555c3b364cf3b9b582ee3e273f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.210260 28490 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250110 03:07:18.210355 28490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.210419 28132 tablet_copy_client.cc:806] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.210528 28132 tablet_copy_client.cc:670] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.211097 28490 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.211216 28490 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.211433 28132 tablet_copy_client.cc:538] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.211833 28490 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.212163 28132 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:18.212630 28490 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:18.216512 28132 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.216667 28132 tablet_bootstrap.cc:492] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:18.216728 28132 ts_tablet_manager.cc:1397] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250110 03:07:18.216809 28132 raft_consensus.cc:357] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.216859 28132 raft_consensus.cc:738] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:18.216879 28132 consensus_queue.cc:260] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.216954 28132 ts_tablet_manager.cc:1428] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.216996 28490 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.217130 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-fcde555c3b364cf3b9b582ee3e273f3e received from {username='slave'} at 127.14.19.132:49387
I20250110 03:07:18.217182 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-fcde555c3b364cf3b9b582ee3e273f3e on tablet fcde555c3b364cf3b9b582ee3e273f3e with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:18.217258 28490 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap complete.
I20250110 03:07:18.217456 28490 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250110 03:07:18.217581 28490 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:18.217648 28490 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: LEARNER
I20250110 03:07:18.217675 28490 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:18.217780 28490 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.218010 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session ab3ed3d8c2394f78943461e22853c7ac-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.130:43035
I20250110 03:07:18.218060 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session ab3ed3d8c2394f78943461e22853c7ac-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer ab3ed3d8c2394f78943461e22853c7ac
I20250110 03:07:18.223443 21730 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:18.223630 28024 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.224625 23100 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:18.224895 28023 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.247829 28543 ts_tablet_manager.cc:927] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:18.248021 28543 tablet_copy_client.cc:323] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:18.248199 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 49c3c5c20dcc4474b207b9c73904219f from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:46977)
I20250110 03:07:18.248253 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 49c3c5c20dcc4474b207b9c73904219f from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:46977: session id = 531cf07bcce6435c9e18185c60b2118d-49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:18.248610 21488 tablet_copy_source_session.cc:215] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.248916 28543 ts_tablet_manager.cc:1905] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.250708 28543 ts_tablet_manager.cc:1918] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20250110 03:07:18.250805 28543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49c3c5c20dcc4474b207b9c73904219f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.251488 28543 tablet_copy_client.cc:806] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.251672 28543 tablet_copy_client.cc:670] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.252343 28543 tablet_copy_client.cc:538] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.252923 28543 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:18.257584 28543 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.257823 28543 tablet_bootstrap.cc:492] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:18.257927 28543 ts_tablet_manager.cc:1397] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20250110 03:07:18.258025 28543 raft_consensus.cc:357] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.258107 28543 raft_consensus.cc:738] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:18.258149 28543 consensus_queue.cc:260] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.258369 28543 ts_tablet_manager.cc:1428] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:18.258405 28405 ts_tablet_manager.cc:927] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:18.258540 28405 tablet_copy_client.cc:323] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:18.258709 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-49c3c5c20dcc4474b207b9c73904219f received from {username='slave'} at 127.14.19.134:46977
I20250110 03:07:18.258757 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-49c3c5c20dcc4474b207b9c73904219f on tablet 49c3c5c20dcc4474b207b9c73904219f with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:18.258738 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41db68f089644324af518422d2cad2fc from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:18.258806 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41db68f089644324af518422d2cad2fc from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-41db68f089644324af518422d2cad2fc
I20250110 03:07:18.259354 21620 tablet_copy_source_session.cc:215] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.259621 28405 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.260943 28405 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250110 03:07:18.261075 28405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.261605 28405 tablet_copy_client.cc:806] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.261703 28405 tablet_copy_client.cc:670] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.262306 28405 tablet_copy_client.cc:538] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.262859 28405 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:18.265493 28405 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.265664 28405 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:18.265744 28405 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:18.265837 28405 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.265894 28405 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:18.265918 28405 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.266005 28405 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.266228 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-41db68f089644324af518422d2cad2fc received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:18.266288 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-41db68f089644324af518422d2cad2fc on tablet 41db68f089644324af518422d2cad2fc with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:18.266754 28543 ts_tablet_manager.cc:927] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:18.266865 28543 tablet_copy_client.cc:323] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:18.267043 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 250f547c54a84a35b2850d50b897810e from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:38517)
I20250110 03:07:18.267103 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 250f547c54a84a35b2850d50b897810e from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:38517: session id = 531cf07bcce6435c9e18185c60b2118d-250f547c54a84a35b2850d50b897810e
I20250110 03:07:18.267575 21358 tablet_copy_source_session.cc:215] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.267850 28543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.268617 28132 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:18.268750 28132 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:18.268879 28543 tablet_copy_client.cc:806] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.268958 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:56489)
I20250110 03:07:18.268998 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:56489: session id = 9da9d601b7d841b5a8f2c2c53f62a951-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:18.269001 28543 tablet_copy_client.cc:670] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.269349 21620 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.269625 28132 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.269836 28543 tablet_copy_client.cc:538] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.270486 28543 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:18.270738 24726 raft_consensus.cc:988] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: : Instructing follower e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:18.270795 24726 raft_consensus.cc:1076] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Signalling peer e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:18.270965 22968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "250f547c54a84a35b2850d50b897810e"
dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
from {username='slave'} at 127.14.19.129:48197
I20250110 03:07:18.271034 28132 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.12
I20250110 03:07:18.271054 22968 raft_consensus.cc:491] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:18.271088 22968 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.271121 28132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.271607 28132 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.271715 28132 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.271838 22968 raft_consensus.cc:513] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.272019 22968 leader_election.cc:290] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 2 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:18.272209 21598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250f547c54a84a35b2850d50b897810e" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:18.272287 21598 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.272250 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "250f547c54a84a35b2850d50b897810e" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:18.272320 21335 raft_consensus.cc:3049] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:18.272341 21335 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:18.272392 21335 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.272505 21335 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.273058 21598 raft_consensus.cc:2463] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:18.273239 21335 raft_consensus.cc:2463] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:18.273273 22905 leader_election.cc:304] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c57c9dfa476f4d21b79f81114c5d74cb, e4d83809ce4746d5ad65e8462857fd9c; no voters:
I20250110 03:07:18.273365 28291 raft_consensus.cc:2798] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:18.273415 28291 raft_consensus.cc:695] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Becoming Leader. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Running, Role: LEADER
I20250110 03:07:18.273487 28291 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.274509 28132 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.274626 21194 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: term changed from 1 to 2, leader changed from a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) to e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.275174 28132 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:18.276196 28543 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.276405 28543 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:18.276500 28543 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20250110 03:07:18.276592 28543 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.276664 28543 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:18.276698 28543 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.276815 28543 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.278766 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-250f547c54a84a35b2850d50b897810e received from {username='slave'} at 127.14.19.134:38517
I20250110 03:07:18.278841 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-250f547c54a84a35b2850d50b897810e on tablet 250f547c54a84a35b2850d50b897810e with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:18.279256 28132 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.279458 28132 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:18.279560 28132 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20250110 03:07:18.279675 28132 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.279742 28132 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:18.279776 28132 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.279875 28132 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.280103 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.132:56489
I20250110 03:07:18.280145 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:18.297587 28132 ts_tablet_manager.cc:927] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:18.297695 28132 tablet_copy_client.cc:323] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:18.297881 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:42117)
I20250110 03:07:18.297926 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:42117: session id = 9da9d601b7d841b5a8f2c2c53f62a951-7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:18.298281 21358 tablet_copy_source_session.cc:215] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.298494 28132 ts_tablet_manager.cc:1905] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:18.299826 28132 ts_tablet_manager.cc:1918] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20250110 03:07:18.299907 28132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.300402 28132 tablet_copy_client.cc:806] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.300489 28132 tablet_copy_client.cc:670] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.301034 28132 tablet_copy_client.cc:538] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.301529 28132 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:18.303862 28184 raft_consensus.cc:988] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: : Instructing follower e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:18.303925 28184 raft_consensus.cc:1076] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Signalling peer e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:18.304108 22968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
from {username='slave'} at 127.14.19.131:59467
I20250110 03:07:18.304184 22968 raft_consensus.cc:491] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:18.304211 22968 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.304668 28132 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.304860 28132 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:18.304952 28132 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:18.304952 22968 raft_consensus.cc:513] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:18.305052 28132 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.305119 28132 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:18.305119 22968 leader_election.cc:290] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 2 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351)
I20250110 03:07:18.305150 28132 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.305235 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:18.305264 28132 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20250110 03:07:18.305284 21730 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.305459 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-7dbef1612c8b4e2baa670b1cbe0e8960 received from {username='slave'} at 127.14.19.132:42117
I20250110 03:07:18.305509 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-7dbef1612c8b4e2baa670b1cbe0e8960 on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:18.305919 21730 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:18.306017 21598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:18.306104 21598 raft_consensus.cc:3049] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:18.306124 21598 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:18.306170 21598 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:18.306281 21598 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.306790 21598 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:18.306995 22905 leader_election.cc:304] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, c57c9dfa476f4d21b79f81114c5d74cb, e4d83809ce4746d5ad65e8462857fd9c; no voters:
I20250110 03:07:18.307137 28293 raft_consensus.cc:2798] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:18.307199 28293 raft_consensus.cc:695] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Becoming Leader. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Running, Role: LEADER
I20250110 03:07:18.307264 28293 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:18.307775 21194 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: term changed from 1 to 2, leader changed from c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) to e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:18.309787 23100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "531cf07bcce6435c9e18185c60b2118d"
I20250110 03:07:18.309856 23100 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:18.310343 23100 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 2.
I20250110 03:07:18.392405 28392 ts_tablet_manager.cc:927] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:18.392565 28392 tablet_copy_client.cc:323] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:18.392774 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet f1e7baec841a4b7e9cb933d786ec2552 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:18.392843 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet f1e7baec841a4b7e9cb933d786ec2552 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:18.393257 21620 tablet_copy_source_session.cc:215] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:18.393630 28392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e7baec841a4b7e9cb933d786ec2552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:18.394657 28392 tablet_copy_client.cc:806] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:18.394793 28392 tablet_copy_client.cc:670] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:18.395486 28392 tablet_copy_client.cc:538] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:18.396020 28392 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:18.399377 28392 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:18.399585 28392 tablet_bootstrap.cc:492] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:18.399684 28392 ts_tablet_manager.cc:1397] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250110 03:07:18.399804 28392 raft_consensus.cc:357] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:18.399864 28392 raft_consensus.cc:738] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:18.399886 28392 consensus_queue.cc:260] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:18.399971 28392 ts_tablet_manager.cc:1428] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:18.400170 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-f1e7baec841a4b7e9cb933d786ec2552 received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:18.400235 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-f1e7baec841a4b7e9cb933d786ec2552 on tablet f1e7baec841a4b7e9cb933d786ec2552 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:18.419726 21730 raft_consensus.cc:1212] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:18.431845 22968 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:18.432399 23229 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:18.432464 23364 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:18.432593 28293 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:18.432691 28291 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:18.433383 28291 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:18.433526 23229 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:18.434538 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0760e9b8e3c642f09c3287ca586a2a1d with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.434608 21194 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 11 to 13, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.434866 23364 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:18.435560 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:18.435753 28391 tablet_replica.cc:331] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:18.435820 28391 raft_consensus.cc:2238] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.435878 28391 raft_consensus.cc:2267] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.436410 28391 ts_tablet_manager.cc:1905] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.437744 28391 ts_tablet_manager.cc:1918] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250110 03:07:18.437804 28391 log.cc:1198] T 0760e9b8e3c642f09c3287ca586a2a1d P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:18.438153 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet 0760e9b8e3c642f09c3287ca586a2a1d (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:18.627409 21335 raft_consensus.cc:1212] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20250110 03:07:18.652622 23364 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:18.653041 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:18.653071 23100 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:18.653249 28023 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:18.653479 28023 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:18.653940 28024 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.654176 23229 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.654699 23100 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.655186 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ea70c2cc77384698aedb06bee2b639ce with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.655418 21189 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 11 to 13, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:18.656358 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet ea70c2cc77384698aedb06bee2b639ce with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:18.656553 28681 tablet_replica.cc:331] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:18.656616 28681 raft_consensus.cc:2238] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.656662 28681 raft_consensus.cc:2267] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.656885 28681 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.658078 28681 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250110 03:07:18.658125 28681 log.cc:1198] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:18.658463 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet ea70c2cc77384698aedb06bee2b639ce (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:18.667188 21335 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.667419 28293 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.668385 23100 raft_consensus.cc:3054] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Advancing to term 2
I20250110 03:07:18.669113 23100 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.669265 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.669538 28293 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.669643 24788 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.670854 28293 raft_consensus.cc:1059] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:18.671025 28293 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:18.671505 23100 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:18.671533 21335 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:18.671556 23229 raft_consensus.cc:1212] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20250110 03:07:18.671777 28291 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.671849 24788 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.672724 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:18.673008 28291 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.672992 28293 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 13 to 15, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.673192 21335 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.673446 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.674516 21189 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 13 to 15, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:18.676003 23100 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:18.676720 21730 raft_consensus.cc:1212] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20250110 03:07:18.693766 22968 raft_consensus.cc:1212] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250110 03:07:18.695329 21730 raft_consensus.cc:1212] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20250110 03:07:18.699661 25088 raft_consensus.cc:1059] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:18.699820 25088 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:18.700315 23229 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:18.700316 21335 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:18.700492 23364 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:18.700500 28701 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:18.700676 28701 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:18.700768 25088 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:18.702236 25088 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.703423 23229 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.703646 23364 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.703642 21188 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 10 to 11, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:18.704027 21335 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.706168 21467 consensus_queue.cc:237] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:18.706671 23229 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:18.706720 21335 raft_consensus.cc:1270] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:18.706869 28768 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.706990 28768 consensus_queue.cc:1035] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:18.707615 28701 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.707728 21335 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.708715 23229 raft_consensus.cc:2949] T a790eedb619d47c6a3ba7eba194b70c1 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.708983 21194 catalog_manager.cc:5526] T a790eedb619d47c6a3ba7eba194b70c1 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:18.709796 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet a790eedb619d47c6a3ba7eba194b70c1 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:18.710021 28777 tablet_replica.cc:331] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:18.710048 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a790eedb619d47c6a3ba7eba194b70c1 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.710090 28777 raft_consensus.cc:2238] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.710235 28777 raft_consensus.cc:2267] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.710829 28777 ts_tablet_manager.cc:1905] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.712956 28777 ts_tablet_manager.cc:1918] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:18.713016 28777 log.cc:1198] T a790eedb619d47c6a3ba7eba194b70c1 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:18.713392 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet a790eedb619d47c6a3ba7eba194b70c1 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:18.723304 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:18.724530 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.724768 28702 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.728123 23229 raft_consensus.cc:1212] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20250110 03:07:18.729759 21318 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:18.730657 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:18.730960 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:18.731463 23344 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:18.734804 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:18.734915 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:18.737293 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:18.756325 23364 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:18.756745 23100 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.756754 23229 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20250110 03:07:18.756896 23300 consensus_peers.cc:487] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.756950 28021 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.757046 28024 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.757624 28024 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:18.758533 23100 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:18.758781 21194 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.760119 23229 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:18.763088 21730 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:18.763646 21335 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:18.763676 22968 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:18.763706 21664 consensus_peers.cc:487] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b16001c451b455f9eda42f3c98a2fdd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.763846 28812 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:18.763957 27286 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:18.764657 28812 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:18.764765 22968 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:18.764786 21335 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:18.765874 21194 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 6 to 7, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.769583 21730 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:18.769974 23364 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:18.769989 21598 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20250110 03:07:18.770120 21665 consensus_peers.cc:487] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): Couldn't send request to peer 531cf07bcce6435c9e18185c60b2118d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.770161 28812 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:18.770248 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:18.770851 28812 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:18.770961 23364 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:18.770954 21598 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:18.771955 21190 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 9 to 10, NON_VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) added. New cstate: current_term: 2 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } } }
I20250110 03:07:18.774893 21467 raft_consensus.cc:1212] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 LEARNER]: Deduplicated request from leader. Original: 2.16->[2.17-2.17] Dedup: 2.17->[]
I20250110 03:07:18.776180 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:18.776614 23229 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.776683 21730 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.776738 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20250110 03:07:18.776887 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.776890 28825 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.777007 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.777115 28184 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.777571 24437 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.777680 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.778610 24437 raft_consensus.cc:1059] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:18.778777 24437 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:18.778807 21730 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.779168 21196 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.779696 21729 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:18.779987 28680 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:18.780099 23231 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
W20250110 03:07:18.780371 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41db68f089644324af518422d2cad2fc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.780293 23229 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.780457 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:18.780640 28680 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:18.781258 28825 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.781400 28680 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.781370 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.781867 23229 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.782709 21729 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.782897 21187 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.783494 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.783694 28702 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.785207 22968 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:18.785321 21598 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:18.785614 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:18.785693 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:18.785871 28293 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.785938 28291 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.786471 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.786522 24788 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:18.786669 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:18.787015 23100 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:18.787025 21335 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:18.787190 28680 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.787245 28184 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.787535 23364 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:18.787635 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.787838 28184 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:18.787969 21190 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 13 to 15, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:18.788121 28680 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.788915 23364 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.789006 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:18.789206 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.790172 24437 raft_consensus.cc:1059] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:18.790395 21196 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 13 to 14, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.790771 21335 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.790836 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 13 to 15, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
W20250110 03:07:18.791047 21535 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.791186 28838 tablet_replica.cc:331] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:18.791224 28838 raft_consensus.cc:2238] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.791265 28838 raft_consensus.cc:2267] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.791464 28838 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.792023 21730 raft_consensus.cc:1212] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20250110 03:07:18.792166 24437 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.792675 23100 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:18.792702 21335 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20250110 03:07:18.792937 21535 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.793002 28846 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.793061 23364 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:18.793198 28825 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.793416 28825 consensus_queue.cc:1037] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 1.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:18.793771 28184 raft_consensus.cc:1022] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: attempt to promote peer a2b4fdb2c41d4a7ebf55cfac417c3422: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:18.793990 28827 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.794153 21335 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.794332 23364 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.795403 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.795578 21190 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.796299 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:18.796794 21335 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20250110 03:07:18.796813 21535 consensus_peers.cc:487] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.796983 23100 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.796996 28846 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.797165 28827 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.797463 21467 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:18.797648 24437 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.797953 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
W20250110 03:07:18.797986 21403 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.798177 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.798327 28768 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.798511 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.798524 25088 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.798720 21335 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.798930 21196 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.799196 28768 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.799328 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.799829 23100 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:18.800521 28838 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:18.800608 28838 log.cc:1198] T 81b3bd116ad54a9fa10d2d0c829bb467 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:18.800617 21190 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.800995 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:18.801105 28777 tablet_replica.cc:331] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:18.801175 28777 raft_consensus.cc:2238] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.801189 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 81b3bd116ad54a9fa10d2d0c829bb467 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:18.801283 28777 raft_consensus.cc:2267] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.801622 28777 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.802040 23102 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:18.803125 28777 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20250110 03:07:18.803201 28777 log.cc:1198] T 41fa30056c0a4b30ab8fcaa14645e7a4 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:18.803671 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:18.839068 21318 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:18.839226 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:18.840309 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:18.841828 21580 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20250110 03:07:18.847474 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:18.848011 23102 raft_consensus.cc:1212] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20250110 03:07:18.853142 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:18.853863 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:18.857002 23344 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:18.873594 21730 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:18.874122 22968 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:18.874114 21335 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:18.874189 21664 consensus_peers.cc:487] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b16001c451b455f9eda42f3c98a2fdd. This is attempt 1: this message will repeat every 5th retry.
W20250110 03:07:18.874189 21665 consensus_peers.cc:487] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b16001c451b455f9eda42f3c98a2fdd. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.874364 28811 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:18.874455 27286 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:18.875118 28811 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:18.875273 21335 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:18.875363 22968 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:18.876562 21190 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 7 to 8, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.880805 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:18.881352 21730 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.881434 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.881383 23364 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:18.881557 28843 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.881685 28846 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20250110 03:07:18.881765 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): Couldn't send request to peer ab3ed3d8c2394f78943461e22853c7ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.881865 28846 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:18.882269 28846 raft_consensus.cc:1022] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: attempt to promote peer 9da9d601b7d841b5a8f2c2c53f62a951: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:18.882572 28843 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.882727 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.883798 21730 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.883845 23364 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.883915 21189 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 15 to 16, NON_VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.884975 24437 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:18.885098 24437 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:18.885648 21730 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:18.885779 23364 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:18.885905 28846 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:18.886003 28845 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:18.886143 22968 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:18.886322 28845 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20250110 03:07:18.886391 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): Couldn't send request to peer ab3ed3d8c2394f78943461e22853c7ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.887230 28845 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.887436 21730 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.888384 23364 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.889190 21189 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 16 to 17, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.889832 22968 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.890190 21598 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
W20250110 03:07:18.890789 21535 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb -> Peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): Couldn't send request to peer ab3ed3d8c2394f78943461e22853c7ac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32609d619a0242d1ba0a851fa0a7812d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:18.890973 23364 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:18.891074 21730 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:18.891140 28184 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:18.891382 28184 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:18.892046 28827 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.892639 23364 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.893098 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:18.893119 21730 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } } }
I20250110 03:07:18.893792 21190 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:18.894712 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:18.895467 28957 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:18.895567 28957 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:18.895673 28957 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:18.896230 28957 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:18.897781 28957 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20250110 03:07:18.897876 28957 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:18.898240 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:18.911630 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44870
I20250110 03:07:18.911715 21598 raft_consensus.cc:604] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:18.945611 22968 raft_consensus.cc:1212] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
I20250110 03:07:19.017405 25176 raft_consensus.cc:1059] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:19.017606 25176 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.018132 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:19.018167 23229 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:19.018167 21730 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:19.018381 28593 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:19.018507 24726 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:19.018663 29019 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:19.019479 24726 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.019629 23229 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.020478 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.020748 21189 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.020742 21730 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.075196 28021 raft_consensus.cc:1059] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:19.075426 28021 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:19.075966 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:19.075949 22968 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:19.075987 21598 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:19.076272 28021 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:19.076469 29054 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:19.076761 28024 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:19.077316 29054 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.077443 22968 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.078271 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.078622 21598 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.078606 21190 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 12 to 13, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.079712 23364 consensus_queue.cc:237] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:19.080099 21730 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:19.080289 28024 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.080317 22968 raft_consensus.cc:1270] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:19.080524 28024 consensus_queue.cc:1035] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.080811 28021 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.080922 21730 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.081709 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.081902 22968 raft_consensus.cc:2949] T 44bb3816ff5a4d44a249b8bbabdc7b04 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.081912 21190 catalog_manager.cc:5526] T 44bb3816ff5a4d44a249b8bbabdc7b04 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 13 to 14, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.082616 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 44bb3816ff5a4d44a249b8bbabdc7b04 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:19.082715 28838 tablet_replica.cc:331] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:19.082773 28838 raft_consensus.cc:2238] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.082887 28838 raft_consensus.cc:2267] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.083236 28838 ts_tablet_manager.cc:1905] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.084275 28838 ts_tablet_manager.cc:1918] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250110 03:07:19.084326 28838 log.cc:1198] T 44bb3816ff5a4d44a249b8bbabdc7b04 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:19.084671 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 44bb3816ff5a4d44a249b8bbabdc7b04 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:19.107690 21335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48954
I20250110 03:07:19.107789 21335 raft_consensus.cc:604] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:19.136510 29083 ts_tablet_manager.cc:927] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:19.136704 29083 tablet_copy_client.cc:323] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.136:42591
I20250110 03:07:19.136906 23370 tablet_copy_service.cc:140] P d0be0af85dee41dd8d01ef4b2af7726f: Received BeginTabletCopySession request for tablet ea70c2cc77384698aedb06bee2b639ce from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:34013)
I20250110 03:07:19.136978 23370 tablet_copy_service.cc:161] P d0be0af85dee41dd8d01ef4b2af7726f: Beginning new tablet copy session on tablet ea70c2cc77384698aedb06bee2b639ce from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:34013: session id = 9da9d601b7d841b5a8f2c2c53f62a951-ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:19.137475 23370 tablet_copy_source_session.cc:215] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.137771 29083 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.139372 29083 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.12
I20250110 03:07:19.139477 29083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea70c2cc77384698aedb06bee2b639ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.140161 29083 tablet_copy_client.cc:806] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.140273 29083 tablet_copy_client.cc:670] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.140978 29083 tablet_copy_client.cc:538] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.141681 29083 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:19.144085 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44916
I20250110 03:07:19.144157 21598 raft_consensus.cc:604] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:19.146307 29083 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.146468 29083 tablet_bootstrap.cc:492] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:19.146572 29083 ts_tablet_manager.cc:1397] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20250110 03:07:19.146682 29083 raft_consensus.cc:357] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:19.146732 29083 raft_consensus.cc:738] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:19.146752 29083 consensus_queue.cc:260] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:19.146838 29083 ts_tablet_manager.cc:1428] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.147045 23370 tablet_copy_service.cc:342] P d0be0af85dee41dd8d01ef4b2af7726f: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-ea70c2cc77384698aedb06bee2b639ce received from {username='slave'} at 127.14.19.132:34013
I20250110 03:07:19.147100 23370 tablet_copy_service.cc:434] P d0be0af85dee41dd8d01ef4b2af7726f: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-ea70c2cc77384698aedb06bee2b639ce on tablet ea70c2cc77384698aedb06bee2b639ce with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:19.177650 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44962
I20250110 03:07:19.177726 21598 raft_consensus.cc:604] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:19.181403 29151 ts_tablet_manager.cc:927] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:19.181595 29151 tablet_copy_client.cc:323] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:19.181787 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet b5064ab85103488f873fffef55fd78fd from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:33889)
I20250110 03:07:19.181850 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet b5064ab85103488f873fffef55fd78fd from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:33889: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-b5064ab85103488f873fffef55fd78fd
I20250110 03:07:19.182368 21488 tablet_copy_source_session.cc:215] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.182719 29151 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.184221 29151 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.12
I20250110 03:07:19.184310 29151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.184988 29151 tablet_copy_client.cc:806] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.185087 29151 tablet_copy_client.cc:670] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.185669 29151 tablet_copy_client.cc:538] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.186316 29151 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:19.191013 29151 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.191232 29151 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:19.191360 29151 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20250110 03:07:19.191462 29151 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.191545 29151 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:19.191581 29151 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.191702 29151 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.192008 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-b5064ab85103488f873fffef55fd78fd received from {username='slave'} at 127.14.19.129:33889
I20250110 03:07:19.192057 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-b5064ab85103488f873fffef55fd78fd on tablet b5064ab85103488f873fffef55fd78fd with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:19.196621 29174 ts_tablet_manager.cc:927] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:19.196763 29174 tablet_copy_client.cc:323] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:19.196938 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:19.196990 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:19.197461 21620 tablet_copy_source_session.cc:215] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.197744 29174 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.199028 29174 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20250110 03:07:19.199121 29174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41fa30056c0a4b30ab8fcaa14645e7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.199869 29174 tablet_copy_client.cc:806] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.199990 29174 tablet_copy_client.cc:670] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.200651 29174 tablet_copy_client.cc:538] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.201190 29174 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:19.205096 29174 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.205255 29174 tablet_bootstrap.cc:492] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:19.205348 29174 ts_tablet_manager.cc:1397] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250110 03:07:19.205423 29174 raft_consensus.cc:357] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:19.205473 29174 raft_consensus.cc:738] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:19.205493 29174 consensus_queue.cc:260] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:19.205572 29174 ts_tablet_manager.cc:1428] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.205771 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-41fa30056c0a4b30ab8fcaa14645e7a4 received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:19.205816 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-41fa30056c0a4b30ab8fcaa14645e7a4 on tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:19.214766 28827 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER ab3ed3d8c2394f78943461e22853c7ac to VOTER
I20250110 03:07:19.214906 28827 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } }
I20250110 03:07:19.215379 23364 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250110 03:07:19.215399 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250110 03:07:19.215528 21467 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250110 03:07:19.215611 28184 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.215759 28827 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.215844 28184 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.216662 28184 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.18: config changed from index 17 to 18, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.216795 21467 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.217911 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.217942 23364 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.218036 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 17 to 18, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.219059 21598 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } }
I20250110 03:07:19.219461 21467 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:19.219537 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:19.219694 29151 ts_tablet_manager.cc:927] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:19.219705 28827 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.219827 29151 tablet_copy_client.cc:323] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:19.219923 28827 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.220218 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41db68f089644324af518422d2cad2fc from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:51055)
I20250110 03:07:19.220276 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41db68f089644324af518422d2cad2fc from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:51055: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-41db68f089644324af518422d2cad2fc
I20250110 03:07:19.220661 28846 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.19: config changed from index 18 to 19, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.220813 21620 tablet_copy_source_session.cc:215] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.220863 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.221086 29151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.221905 21467 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.221966 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 18 to 19, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.222393 29151 tablet_copy_client.cc:806] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.222492 29151 tablet_copy_client.cc:670] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.222853 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 18: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.223085 22968 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:19.223517 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:19.223598 23102 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:19.223894 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 19) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:19.224004 28777 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:19.224063 28777 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.224037 28293 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:19.224144 28293 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:19.224181 28777 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.224472 29151 tablet_copy_client.cc:538] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.224558 28777 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.225023 29151 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:19.225757 28777 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.18
I20250110 03:07:19.225812 28777 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:19.226230 28702 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.226390 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.227257 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.227340 23102 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.227566 21190 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 15 to 16, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.228257 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:19.228346 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 250f547c54a84a35b2850d50b897810e with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:19.230412 29151 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.230608 29151 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:19.230697 29151 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250110 03:07:19.230803 29193 tablet_replica.cc:331] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:19.230823 29151 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.230868 29193 raft_consensus.cc:2238] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.230904 29151 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:19.230947 29193 raft_consensus.cc:2267] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.230942 29151 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 1.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.231063 29151 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.231223 29193 ts_tablet_manager.cc:1905] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.231460 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-41db68f089644324af518422d2cad2fc received from {username='slave'} at 127.14.19.129:51055
I20250110 03:07:19.231504 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-41db68f089644324af518422d2cad2fc on tablet 41db68f089644324af518422d2cad2fc with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:19.232623 29193 ts_tablet_manager.cc:1918] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250110 03:07:19.232672 29193 log.cc:1198] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/250f547c54a84a35b2850d50b897810e
I20250110 03:07:19.232954 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 250f547c54a84a35b2850d50b897810e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:19.235283 25088 raft_consensus.cc:1059] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:19.235427 25088 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.235857 23364 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:19.235896 21730 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:19.235921 23229 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:19.236097 28767 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:19.236191 28768 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:19.236258 28701 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:19.236944 28768 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.237073 23229 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.237157 23364 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.238034 21190 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 15 to 16, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.238343 21730 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.239495 21467 consensus_queue.cc:237] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.239838 23229 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.239913 21730 raft_consensus.cc:1270] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.240065 28701 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.240142 25088 consensus_queue.cc:1035] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.240648 25088 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.240811 23229 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.240968 21730 raft_consensus.cc:2949] T fcde555c3b364cf3b9b582ee3e273f3e P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.241562 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet fcde555c3b364cf3b9b582ee3e273f3e with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.241739 21189 catalog_manager.cc:5526] T fcde555c3b364cf3b9b582ee3e273f3e P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 16 to 17, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.242504 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet fcde555c3b364cf3b9b582ee3e273f3e with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:19.242626 28777 tablet_replica.cc:331] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:19.242694 28777 raft_consensus.cc:2238] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.242797 28777 raft_consensus.cc:2267] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.242996 28777 ts_tablet_manager.cc:1905] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.244047 28777 ts_tablet_manager.cc:1918] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.16
I20250110 03:07:19.244096 28777 log.cc:1198] T fcde555c3b364cf3b9b582ee3e273f3e P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:19.244402 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet fcde555c3b364cf3b9b582ee3e273f3e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:19.248474 28767 raft_consensus.cc:1059] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:19.248631 28767 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:19.249069 23229 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.249228 22968 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.249243 28701 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.249372 23102 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.249541 28701 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.249625 28768 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.250466 28701 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.250640 22968 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.251562 23229 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.251658 23102 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.251725 21192 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 13 to 14, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.252887 21467 consensus_queue.cc:237] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:19.253237 23229 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:19.253280 23102 raft_consensus.cc:1270] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:19.253520 25088 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:19.253598 28767 consensus_queue.cc:1035] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:19.254268 28767 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.254818 23102 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.255511 23229 raft_consensus.cc:2949] T d52b50e3c87b47bead3a25f7255d1c42 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.255719 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d52b50e3c87b47bead3a25f7255d1c42 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.255980 21189 catalog_manager.cc:5526] T d52b50e3c87b47bead3a25f7255d1c42 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.256742 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet d52b50e3c87b47bead3a25f7255d1c42 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:19.256853 28957 tablet_replica.cc:331] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:19.256911 28957 raft_consensus.cc:2238] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.256980 28957 raft_consensus.cc:2267] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.257220 28957 ts_tablet_manager.cc:1905] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.258409 28957 ts_tablet_manager.cc:1918] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250110 03:07:19.258467 28957 log.cc:1198] T d52b50e3c87b47bead3a25f7255d1c42 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:19.258875 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet d52b50e3c87b47bead3a25f7255d1c42 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:19.293577 29232 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:19.293721 29232 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:19.293915 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer ab3ed3d8c2394f78943461e22853c7ac ({username='slave'} at 127.14.19.130:43035)
I20250110 03:07:19.293962 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer ab3ed3d8c2394f78943461e22853c7ac at {username='slave'} at 127.14.19.130:43035: session id = ab3ed3d8c2394f78943461e22853c7ac-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:19.294291 21620 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.294624 29232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.295509 29232 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.295637 29232 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.296175 29232 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.296716 29232 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap starting.
I20250110 03:07:19.299294 24437 raft_consensus.cc:988] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: : Instructing follower d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:19.299389 24437 raft_consensus.cc:1076] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Signalling peer d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:19.299647 23364 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
from {username='slave'} at 127.14.19.131:44395
I20250110 03:07:19.299724 23364 raft_consensus.cc:491] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:19.299751 23364 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.300609 23364 raft_consensus.cc:513] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:19.300777 23364 leader_election.cc:290] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:19.300897 21598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: true dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:19.300976 21598 raft_consensus.cc:3049] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:19.300999 21598 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:19.301012 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32609d619a0242d1ba0a851fa0a7812d" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:19.301051 21598 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:19.301088 21730 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.301122 21598 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.301631 21598 raft_consensus.cc:2463] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:19.301648 21730 raft_consensus.cc:2463] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:19.301797 23298 leader_election.cc:304] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c57c9dfa476f4d21b79f81114c5d74cb, d0be0af85dee41dd8d01ef4b2af7726f; no voters:
I20250110 03:07:19.301888 28024 raft_consensus.cc:2798] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:19.301937 28024 raft_consensus.cc:695] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Becoming Leader. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Running, Role: LEADER
I20250110 03:07:19.301949 29232 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.301990 28024 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:19.302134 29232 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: Bootstrap complete.
I20250110 03:07:19.302228 29232 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250110 03:07:19.302309 29232 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:19.302363 29232 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab3ed3d8c2394f78943461e22853c7ac, State: Initialized, Role: LEARNER
I20250110 03:07:19.302397 29232 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 1.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } }
I20250110 03:07:19.302497 29232 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.302573 21192 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: term changed from 1 to 2, leader changed from c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) to d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136). New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.302731 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session ab3ed3d8c2394f78943461e22853c7ac-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.130:43035
I20250110 03:07:19.302800 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session ab3ed3d8c2394f78943461e22853c7ac-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer ab3ed3d8c2394f78943461e22853c7ac
I20250110 03:07:19.303304 29235 ts_tablet_manager.cc:927] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:19.303488 29235 tablet_copy_client.cc:323] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:19.303675 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 80152844286545c68c7656ef9bbe8604 from peer 531cf07bcce6435c9e18185c60b2118d ({username='slave'} at 127.14.19.134:56637)
I20250110 03:07:19.303726 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 80152844286545c68c7656ef9bbe8604 from peer 531cf07bcce6435c9e18185c60b2118d at {username='slave'} at 127.14.19.134:56637: session id = 531cf07bcce6435c9e18185c60b2118d-80152844286545c68c7656ef9bbe8604
I20250110 03:07:19.304096 21751 tablet_copy_source_session.cc:215] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.304369 29235 ts_tablet_manager.cc:1905] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.305569 29235 ts_tablet_manager.cc:1918] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20250110 03:07:19.305667 29235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80152844286545c68c7656ef9bbe8604. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.306180 29235 tablet_copy_client.cc:806] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.306341 29235 tablet_copy_client.cc:670] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.306955 29235 tablet_copy_client.cc:538] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.307588 29235 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap starting.
I20250110 03:07:19.314103 29235 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.314397 29235 tablet_bootstrap.cc:492] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Bootstrap complete.
I20250110 03:07:19.314580 29235 ts_tablet_manager.cc:1397] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250110 03:07:19.314735 29235 raft_consensus.cc:357] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:19.314870 29235 raft_consensus.cc:738] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 531cf07bcce6435c9e18185c60b2118d, State: Initialized, Role: LEARNER
I20250110 03:07:19.314961 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.315030 29235 consensus_queue.cc:260] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: true } }
I20250110 03:07:19.315380 29235 ts_tablet_manager.cc:1428] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:19.315595 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session 531cf07bcce6435c9e18185c60b2118d-80152844286545c68c7656ef9bbe8604 received from {username='slave'} at 127.14.19.134:56637
I20250110 03:07:19.315644 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session 531cf07bcce6435c9e18185c60b2118d-80152844286545c68c7656ef9bbe8604 on tablet 80152844286545c68c7656ef9bbe8604 with peer 531cf07bcce6435c9e18185c60b2118d
I20250110 03:07:19.315656 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:19.315874 28680 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.315896 23229 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:19.316038 21335 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:19.316222 28680 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.316612 28680 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.316694 28846 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.316972 23231 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.318024 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.318470 28843 raft_consensus.cc:1059] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:19.318434 21192 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 17 to 18, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.318444 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.318630 28843 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:19.319789 23231 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250110 03:07:19.319751 21335 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.320044 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.320394 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:19.320670 21337 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250110 03:07:19.320684 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250110 03:07:19.320896 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.320935 21337 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f"
dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48984
I20250110 03:07:19.320987 21337 raft_consensus.cc:604] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:19.320992 24437 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.321906 24437 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.322036 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.322844 29270 tablet_replica.cc:331] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:19.322909 29270 raft_consensus.cc:2238] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.323014 23231 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.323139 29270 raft_consensus.cc:2267] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.323333 21192 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.323448 29270 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.324410 21337 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.325083 29270 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20250110 03:07:19.325158 29270 log.cc:1198] T 41db68f089644324af518422d2cad2fc P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/41db68f089644324af518422d2cad2fc
I20250110 03:07:19.325644 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 41db68f089644324af518422d2cad2fc (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:19.325743 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:19.326745 23231 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250110 03:07:19.326802 21337 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250110 03:07:19.326936 28680 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.327028 28845 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.327531 24437 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.327677 23231 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.328330 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 19: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.328596 21189 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 19 to 20, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.328805 21337 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:19.329242 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 20) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:19.329334 28957 tablet_replica.cc:331] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:19.329394 28957 raft_consensus.cc:2238] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.329461 28957 raft_consensus.cc:2267] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.329670 28957 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.330886 28957 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.19
I20250110 03:07:19.330931 28957 log.cc:1198] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/41db68f089644324af518422d2cad2fc
I20250110 03:07:19.331275 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 41db68f089644324af518422d2cad2fc (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:19.350713 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:19.351540 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:19.351963 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:19.356817 21318 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:19.357700 23210 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:19.363673 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:19.366128 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:19.367560 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:19.368113 29019 raft_consensus.cc:1059] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:19.368292 29019 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.368765 23364 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:19.368813 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:19.369001 28593 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:19.369112 24726 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:19.369156 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:19.369414 24726 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:19.370427 29019 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.370590 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.370580 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.371721 23364 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.371708 21189 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 12 to 13, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.373095 21337 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.373469 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.373477 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.373665 28593 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.373888 28593 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.374219 24726 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.374377 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.374675 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.375221 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.375808 21188 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 13 to 14, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:19.378489 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:19.378664 28777 tablet_replica.cc:331] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:19.378722 28777 raft_consensus.cc:2238] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.378801 28777 raft_consensus.cc:2267] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.379110 28777 ts_tablet_manager.cc:1905] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.380558 28777 ts_tablet_manager.cc:1918] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20250110 03:07:19.380617 28777 log.cc:1198] T 7dbef1612c8b4e2baa670b1cbe0e8960 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:19.380966 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:19.386444 28843 raft_consensus.cc:1059] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:19.386651 28843 consensus_queue.cc:237] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:19.387120 23364 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.387236 22968 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.387292 23102 raft_consensus.cc:1270] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.387318 28680 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.387676 28680 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.387801 28680 consensus_queue.cc:1035] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.388473 28680 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.389513 21194 catalog_manager.cc:5526] T f1e7baec841a4b7e9cb933d786ec2552 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 16 to 17, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.389520 23364 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.389600 23102 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.389670 22968 raft_consensus.cc:2949] T f1e7baec841a4b7e9cb933d786ec2552 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:19.391741 21337 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:19.392122 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:19.392163 21730 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:19.392365 28593 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:19.392453 24726 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20250110 03:07:19.392542 21272 consensus_peers.cc:487] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbef1612c8b4e2baa670b1cbe0e8960. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:19.393047 28593 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:19.393203 21730 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:19.393227 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:19.394313 21189 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 14 to 15, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.398362 21598 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.398891 23364 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:19.398911 21467 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:19.399000 21534 consensus_peers.cc:487] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:19.399092 28680 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:19.399173 28845 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:19.399854 28845 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.400038 21467 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.400661 23364 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:19.401582 21188 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 6 to 7, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.405547 21598 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:19.405929 21467 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:19.405994 23231 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
W20250110 03:07:19.406036 21536 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cc5781db40c4148ba1889c45405efc2. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:19.406172 28680 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.406287 28680 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.407083 28680 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:19.407233 23231 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:19.407975 21467 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:19.409018 21188 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:19.413226 21467 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.413743 23102 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:19.413829 21337 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:19.413976 28768 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20250110 03:07:19.414067 21403 consensus_peers.cc:487] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:19.414454 28768 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:19.414732 28768 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.414933 23102 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.415277 21337 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.416191 21189 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 8 to 9, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.419921 28593 raft_consensus.cc:988] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: : Instructing follower b2e7bce763e24b63bc760b0b82b53454 to start an election
I20250110 03:07:19.419988 28593 raft_consensus.cc:1076] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Signalling peer b2e7bce763e24b63bc760b0b82b53454 to start an election
I20250110 03:07:19.420357 23231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f"
dest_uuid: "b2e7bce763e24b63bc760b0b82b53454"
from {username='slave'} at 127.14.19.129:57051
I20250110 03:07:19.420441 23231 raft_consensus.cc:491] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:19.420470 23231 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.421228 23231 raft_consensus.cc:513] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.421532 23231 leader_election.cc:290] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [CANDIDATE]: Term 2 election: Requested vote from peers a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:19.421703 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "b2e7bce763e24b63bc760b0b82b53454" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:19.421789 21730 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.421996 21337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "b2e7bce763e24b63bc760b0b82b53454" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:19.422070 21337 raft_consensus.cc:3049] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:19.422091 21337 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Running, Role: LEADER
I20250110 03:07:19.422137 21337 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.422210 21337 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.422501 21730 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e7bce763e24b63bc760b0b82b53454 in term 2.
I20250110 03:07:19.422923 21337 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e7bce763e24b63bc760b0b82b53454 in term 2.
I20250110 03:07:19.423091 23167 leader_election.cc:304] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9da9d601b7d841b5a8f2c2c53f62a951, a2b4fdb2c41d4a7ebf55cfac417c3422, b2e7bce763e24b63bc760b0b82b53454; no voters:
I20250110 03:07:19.423209 24800 raft_consensus.cc:2798] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:19.423338 24800 raft_consensus.cc:695] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEADER]: Becoming Leader. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Running, Role: LEADER
I20250110 03:07:19.423401 24800 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:19.423961 21337 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.424031 21192 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: term changed from 1 to 2, leader changed from a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) to b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135). New cstate: current_term: 2 leader_uuid: "b2e7bce763e24b63bc760b0b82b53454" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.424562 23231 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:19.424660 22968 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:19.424739 28593 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250110 03:07:19.424872 21272 consensus_peers.cc:487] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:19.424966 28593 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:19.425890 22968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62aacc8deebd43debd76a2c10815da2f" candidate_uuid: "b2e7bce763e24b63bc760b0b82b53454" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
I20250110 03:07:19.425987 22968 raft_consensus.cc:3054] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:19.426206 28593 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.426693 22968 raft_consensus.cc:2463] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e7bce763e24b63bc760b0b82b53454 in term 2.
I20250110 03:07:19.427345 22968 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.427520 21192 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 9 to 10, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:19.429026 23231 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:19.451900 29418 ts_tablet_manager.cc:927] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:19.452083 29418 tablet_copy_client.cc:323] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:19.452291 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 9b16001c451b455f9eda42f3c98a2fdd from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:41729)
I20250110 03:07:19.452363 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 9b16001c451b455f9eda42f3c98a2fdd from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:41729: session id = d0be0af85dee41dd8d01ef4b2af7726f-9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:19.452900 21751 tablet_copy_source_session.cc:215] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.453233 29418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b16001c451b455f9eda42f3c98a2fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.453658 28767 raft_consensus.cc:1059] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:19.453987 28767 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:19.454990 29418 tablet_copy_client.cc:806] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.455082 21337 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.455113 29418 tablet_copy_client.cc:670] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.455191 23102 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.455426 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.455648 29418 tablet_copy_client.cc:538] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.455705 22968 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:19.455749 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.456209 25088 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:19.456605 29418 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:19.456799 25088 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.456948 22968 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.457769 21337 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.457957 23102 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.458926 21188 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 13 to 14, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.460413 21467 consensus_queue.cc:237] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:19.461339 23102 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:19.461373 21337 raft_consensus.cc:1270] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:19.461567 29418 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.461568 28701 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:19.461638 28767 consensus_queue.cc:1035] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:19.461757 29418 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:19.461858 29418 ts_tablet_manager.cc:1397] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20250110 03:07:19.461972 29418 raft_consensus.cc:357] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.462051 29418 raft_consensus.cc:738] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:19.462102 29418 consensus_queue.cc:260] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.462261 28768 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.462355 29418 ts_tablet_manager.cc:1428] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20250110 03:07:19.462378 21337 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.462527 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-9b16001c451b455f9eda42f3c98a2fdd received from {username='slave'} at 127.14.19.136:41729
I20250110 03:07:19.462608 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-9b16001c451b455f9eda42f3c98a2fdd on tablet 9b16001c451b455f9eda42f3c98a2fdd with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:19.462977 23102 raft_consensus.cc:2949] T 49c3c5c20dcc4474b207b9c73904219f P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.463304 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49c3c5c20dcc4474b207b9c73904219f with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.463619 21192 catalog_manager.cc:5526] T 49c3c5c20dcc4474b207b9c73904219f P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 14 to 15, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:19.464674 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 49c3c5c20dcc4474b207b9c73904219f with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:19.464774 28957 tablet_replica.cc:331] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:19.464835 28957 raft_consensus.cc:2238] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.464903 28957 raft_consensus.cc:2267] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.465131 28957 ts_tablet_manager.cc:1905] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.466192 28957 ts_tablet_manager.cc:1918] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250110 03:07:19.466240 28957 log.cc:1198] T 49c3c5c20dcc4474b207b9c73904219f P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:19.466521 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 49c3c5c20dcc4474b207b9c73904219f (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:19.491050 29465 ts_tablet_manager.cc:927] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: Initiating tablet copy from peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:19.491209 29465 tablet_copy_client.cc:323] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.132:45355
I20250110 03:07:19.491441 21751 tablet_copy_service.cc:140] P 9da9d601b7d841b5a8f2c2c53f62a951: Received BeginTabletCopySession request for tablet 9b16001c451b455f9eda42f3c98a2fdd from peer c57c9dfa476f4d21b79f81114c5d74cb ({username='slave'} at 127.14.19.131:59913)
I20250110 03:07:19.491495 21751 tablet_copy_service.cc:161] P 9da9d601b7d841b5a8f2c2c53f62a951: Beginning new tablet copy session on tablet 9b16001c451b455f9eda42f3c98a2fdd from peer c57c9dfa476f4d21b79f81114c5d74cb at {username='slave'} at 127.14.19.131:59913: session id = c57c9dfa476f4d21b79f81114c5d74cb-9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:19.491981 21751 tablet_copy_source_session.cc:215] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.492300 29465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b16001c451b455f9eda42f3c98a2fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.493268 29465 tablet_copy_client.cc:806] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.493396 29465 tablet_copy_client.cc:670] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.493968 29465 tablet_copy_client.cc:538] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.494624 29465 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:19.497484 29465 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.497692 29465 tablet_bootstrap.cc:492] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap complete.
I20250110 03:07:19.497792 29465 ts_tablet_manager.cc:1397] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:19.497879 29465 raft_consensus.cc:357] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.497967 29465 raft_consensus.cc:738] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: LEARNER
I20250110 03:07:19.498003 29465 consensus_queue.cc:260] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.498135 29465 ts_tablet_manager.cc:1428] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.498294 21751 tablet_copy_service.cc:342] P 9da9d601b7d841b5a8f2c2c53f62a951: Request end of tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-9b16001c451b455f9eda42f3c98a2fdd received from {username='slave'} at 127.14.19.131:59913
I20250110 03:07:19.498353 21751 tablet_copy_service.cc:434] P 9da9d601b7d841b5a8f2c2c53f62a951: ending tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-9b16001c451b455f9eda42f3c98a2fdd on tablet 9b16001c451b455f9eda42f3c98a2fdd with peer c57c9dfa476f4d21b79f81114c5d74cb
I20250110 03:07:19.582967 23231 raft_consensus.cc:1212] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
I20250110 03:07:19.723337 21730 raft_consensus.cc:1212] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20250110 03:07:19.725925 21337 raft_consensus.cc:1212] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.15->[1.16-1.16] Dedup: 1.16->[]
I20250110 03:07:19.747197 23102 raft_consensus.cc:1212] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250110 03:07:19.813459 29600 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:19.813614 29600 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:19.813797 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:56489)
I20250110 03:07:19.813858 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:56489: session id = 9da9d601b7d841b5a8f2c2c53f62a951-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:19.814224 21620 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.814522 29600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.815611 29600 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.815721 29600 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.816321 29600 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.816883 29600 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:19.820642 29600 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.820811 29600 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:19.820899 29600 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250110 03:07:19.821000 29600 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:19.821061 29600 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:19.821094 29600 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 2.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:19.821195 29600 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.821435 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.132:56489
I20250110 03:07:19.821502 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:19.851385 21467 raft_consensus.cc:3054] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEARNER]: Advancing to term 2
I20250110 03:07:19.852079 21467 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:19.852308 28023 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.853066 28023 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER ab3ed3d8c2394f78943461e22853c7ac to VOTER
W20250110 03:07:19.853127 28023 raft_consensus.cc:1062] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Unable to promote non-voter ab3ed3d8c2394f78943461e22853c7ac: Illegal state: Leader has not yet committed an operation in its own term
I20250110 03:07:19.861502 21730 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:19.861719 28023 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.862883 21598 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:19.863137 28023 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:19.864190 28023 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER ab3ed3d8c2394f78943461e22853c7ac to VOTER
I20250110 03:07:19.864344 28023 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 18, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } }
I20250110 03:07:19.864768 21467 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:19.864795 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:19.864827 21598 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:19.864967 29609 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.865051 28024 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.865178 28023 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:19.865661 29610 ts_tablet_manager.cc:927] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:19.865804 29610 tablet_copy_client.cc:323] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:19.865991 28023 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.20: config changed from index 18 to 20, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.866050 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34443)
I20250110 03:07:19.866120 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34443: session id = e4d83809ce4746d5ad65e8462857fd9c-7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:19.866652 21358 tablet_copy_source_session.cc:215] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.866711 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 18 to 20, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.866926 21598 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 18 to 20, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.867257 21194 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 18 to 20, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.868184 23364 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 2.20, Last appended by leader: 18, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } }
I20250110 03:07:19.868211 21467 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 18 to 20, ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.868553 29610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbef1612c8b4e2baa670b1cbe0e8960. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.869123 21468 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:19.869385 29605 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:19.869737 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:19.869884 29465 ts_tablet_manager.cc:927] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:19.869902 29605 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:19.870033 29465 tablet_copy_client.cc:323] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:19.870029 29605 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.870188 21468 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.870249 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 87f8afb021a7407089c710f6dd588d03 from peer c57c9dfa476f4d21b79f81114c5d74cb ({username='slave'} at 127.14.19.131:33069)
I20250110 03:07:19.870296 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 87f8afb021a7407089c710f6dd588d03 from peer c57c9dfa476f4d21b79f81114c5d74cb at {username='slave'} at 127.14.19.131:33069: session id = c57c9dfa476f4d21b79f81114c5d74cb-87f8afb021a7407089c710f6dd588d03
I20250110 03:07:19.870481 29610 tablet_copy_client.cc:806] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.870620 29610 tablet_copy_client.cc:670] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.870769 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.870898 21488 tablet_copy_source_session.cc:215] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.871091 29610 tablet_copy_client.cc:538] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.871203 29465 ts_tablet_manager.cc:1905] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.871913 29610 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:19.871958 21194 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } }
I20250110 03:07:19.872239 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.872684 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 21) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:19.872776 29465 ts_tablet_manager.cc:1918] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20250110 03:07:19.872846 29617 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:19.872874 29465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87f8afb021a7407089c710f6dd588d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.872908 29617 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.873000 29617 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.873557 29465 tablet_copy_client.cc:806] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.873660 29465 tablet_copy_client.cc:670] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.873975 29617 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.874665 29465 tablet_copy_client.cc:538] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.875339 29465 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:19.875644 29617 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.20
I20250110 03:07:19.875743 29617 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:19.876222 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:19.877195 29610 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.877416 29610 tablet_bootstrap.cc:492] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:19.877504 29610 ts_tablet_manager.cc:1397] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250110 03:07:19.877611 29610 raft_consensus.cc:357] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:19.877673 29610 raft_consensus.cc:738] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:19.877708 29610 consensus_queue.cc:260] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:19.877808 29610 ts_tablet_manager.cc:1428] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.878201 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-7dbef1612c8b4e2baa670b1cbe0e8960 received from {username='slave'} at 127.14.19.133:34443
I20250110 03:07:19.878254 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-7dbef1612c8b4e2baa670b1cbe0e8960 on tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:19.880468 29465 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.880661 29465 tablet_bootstrap.cc:492] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap complete.
I20250110 03:07:19.880746 29465 ts_tablet_manager.cc:1397] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250110 03:07:19.880848 29465 raft_consensus.cc:357] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.880908 29465 raft_consensus.cc:738] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: LEARNER
I20250110 03:07:19.880939 29465 consensus_queue.cc:260] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.881031 29465 ts_tablet_manager.cc:1428] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.881255 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-87f8afb021a7407089c710f6dd588d03 received from {username='slave'} at 127.14.19.131:33069
I20250110 03:07:19.881314 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-87f8afb021a7407089c710f6dd588d03 on tablet 87f8afb021a7407089c710f6dd588d03 with peer c57c9dfa476f4d21b79f81114c5d74cb
I20250110 03:07:19.926903 29657 ts_tablet_manager.cc:927] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:19.927044 29657 tablet_copy_client.cc:323] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:19.927232 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet c5d0922741634918830ee2c163cbce24 from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:51055)
I20250110 03:07:19.927280 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet c5d0922741634918830ee2c163cbce24 from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:51055: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-c5d0922741634918830ee2c163cbce24
I20250110 03:07:19.927670 21620 tablet_copy_source_session.cc:215] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.928078 29657 ts_tablet_manager.cc:1905] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.929608 29657 ts_tablet_manager.cc:1918] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250110 03:07:19.929713 29657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.930341 29657 tablet_copy_client.cc:806] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.930446 29657 tablet_copy_client.cc:670] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.931362 29657 tablet_copy_client.cc:538] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.932070 29657 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:19.935994 29657 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.936213 29657 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:19.936321 29657 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250110 03:07:19.936432 29657 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.936523 29657 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:19.936554 29657 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:19.936832 29657 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:07:19.936996 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-c5d0922741634918830ee2c163cbce24 received from {username='slave'} at 127.14.19.129:51055
I20250110 03:07:19.937049 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-c5d0922741634918830ee2c163cbce24 on tablet c5d0922741634918830ee2c163cbce24 with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:19.941828 21729 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:19.942076 29364 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:19.959889 29465 ts_tablet_manager.cc:927] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:19.960036 29465 tablet_copy_client.cc:323] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:19.960264 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 5a1a954889a9452f8a8ee55aa0a6f26e from peer c57c9dfa476f4d21b79f81114c5d74cb ({username='slave'} at 127.14.19.131:41927)
I20250110 03:07:19.960325 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 5a1a954889a9452f8a8ee55aa0a6f26e from peer c57c9dfa476f4d21b79f81114c5d74cb at {username='slave'} at 127.14.19.131:41927: session id = c57c9dfa476f4d21b79f81114c5d74cb-5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:19.960683 21358 tablet_copy_source_session.cc:215] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:19.960932 29465 ts_tablet_manager.cc:1905] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:19.962154 29465 ts_tablet_manager.cc:1918] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20250110 03:07:19.962239 29465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1a954889a9452f8a8ee55aa0a6f26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:19.962843 29465 tablet_copy_client.cc:806] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:19.962951 29465 tablet_copy_client.cc:670] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:19.963548 29465 tablet_copy_client.cc:538] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:19.964175 29465 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:19.966753 29465 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:19.966928 29465 tablet_bootstrap.cc:492] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap complete.
I20250110 03:07:19.967001 29465 ts_tablet_manager.cc:1397] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:07:19.967093 29465 raft_consensus.cc:357] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.967144 29465 raft_consensus.cc:738] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: LEARNER
I20250110 03:07:19.967165 29465 consensus_queue.cc:260] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:19.967247 29465 ts_tablet_manager.cc:1428] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:19.967458 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-5a1a954889a9452f8a8ee55aa0a6f26e received from {username='slave'} at 127.14.19.131:41927
I20250110 03:07:19.967509 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-5a1a954889a9452f8a8ee55aa0a6f26e on tablet 5a1a954889a9452f8a8ee55aa0a6f26e with peer c57c9dfa476f4d21b79f81114c5d74cb
I20250110 03:07:19.969167 28680 raft_consensus.cc:1059] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:19.969308 28680 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:19.969767 23231 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.969811 21337 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.969813 23102 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:19.970005 29709 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.970134 28680 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.970240 28843 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:19.971015 28843 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.971136 23102 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.971946 21337 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.972084 21194 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:19.972805 23231 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.973104 21598 consensus_queue.cc:237] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:19.973809 23231 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:19.973845 21337 raft_consensus.cc:1270] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:19.973979 28680 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.974056 28846 consensus_queue.cc:1035] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:19.974625 28846 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.974803 21337 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.974812 23231 raft_consensus.cc:2949] T 41fa30056c0a4b30ab8fcaa14645e7a4 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.975633 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:19.975847 21192 catalog_manager.cc:5526] T 41fa30056c0a4b30ab8fcaa14645e7a4 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 17 to 18, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:19.976637 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 41fa30056c0a4b30ab8fcaa14645e7a4 with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:19.976821 29713 tablet_replica.cc:331] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:19.976898 29713 raft_consensus.cc:2238] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:19.977015 29713 raft_consensus.cc:2267] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:19.977234 29713 ts_tablet_manager.cc:1905] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:19.978363 29713 ts_tablet_manager.cc:1918] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20250110 03:07:19.978415 29713 log.cc:1198] T 41fa30056c0a4b30ab8fcaa14645e7a4 P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:19.978796 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 41fa30056c0a4b30ab8fcaa14645e7a4 (table rebalance_test_table_7 [id=f3ab21f0e11b41cda6fe4adda29e8028]) successfully deleted
I20250110 03:07:20.010915 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:20.011148 29364 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.012353 21337 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:20.012794 29364 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.013257 23231 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:20.013705 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:20.013785 21729 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:20.013935 29364 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:20.014019 29717 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:20.014613 29364 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.014842 21729 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.015641 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 62aacc8deebd43debd76a2c10815da2f with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.015919 21192 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 11 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "b2e7bce763e24b63bc760b0b82b53454" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.016175 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.017205 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 62aacc8deebd43debd76a2c10815da2f with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:20.017376 29719 tablet_replica.cc:331] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:20.017486 29719 raft_consensus.cc:2238] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.017591 29719 raft_consensus.cc:2267] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.017802 29719 ts_tablet_manager.cc:1905] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.019099 29719 ts_tablet_manager.cc:1918] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250110 03:07:20.019158 29719 log.cc:1198] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:20.019450 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 62aacc8deebd43debd76a2c10815da2f (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:20.027715 23364 raft_consensus.cc:1212] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:20.056595 21598 raft_consensus.cc:1212] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:20.123378 29599 raft_consensus.cc:1059] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER 531cf07bcce6435c9e18185c60b2118d to VOTER
I20250110 03:07:20.123595 29599 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:20.124143 23364 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:07:20.124169 23102 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:07:20.124200 21598 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:07:20.124404 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.124544 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.124622 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.125916 29599 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.126394 23364 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.126771 23102 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.127529 21598 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.127748 21193 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 10 to 11, 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.129040 21729 consensus_queue.cc:237] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } }
I20250110 03:07:20.129536 23102 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:20.130112 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.131258 21598 raft_consensus.cc:1270] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:20.131448 27286 consensus_queue.cc:1035] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.131584 28811 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.132695 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 80152844286545c68c7656ef9bbe8604 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.132745 23102 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.133697 21598 raft_consensus.cc:2949] T 80152844286545c68c7656ef9bbe8604 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } }
I20250110 03:07:20.133832 21193 catalog_manager.cc:5526] T 80152844286545c68c7656ef9bbe8604 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 11 to 12, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 2 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.135293 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 80152844286545c68c7656ef9bbe8604 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:20.135500 29795 tablet_replica.cc:331] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:20.135562 29795 raft_consensus.cc:2238] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.135644 29795 raft_consensus.cc:2267] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.137111 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:20.137301 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.142526 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.143496 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:20.145931 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.146775 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:20.147095 29795 ts_tablet_manager.cc:1905] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.149196 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:20.149259 29795 ts_tablet_manager.cc:1918] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250110 03:07:20.149315 29795 log.cc:1198] T 80152844286545c68c7656ef9bbe8604 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/80152844286545c68c7656ef9bbe8604
I20250110 03:07:20.149770 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 80152844286545c68c7656ef9bbe8604 (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:20.151227 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.171043 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:20.171550 23231 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
W20250110 03:07:20.171567 21534 consensus_peers.cc:487] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:20.171734 28846 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:20.172360 28843 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:20.173125 21337 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250110 03:07:20.173350 23231 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:20.173549 21193 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 20 to 21, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:20.174415 28843 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:20.175319 21337 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:20.182955 21337 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:20.183394 23231 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:20.183382 23102 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:20.183709 21271 consensus_peers.cc:487] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58e33e63f48f4b758c19774888345ee0. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:20.183706 29019 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:20.183897 29854 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:20.184451 29019 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:20.184595 23102 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:20.184751 23231 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:20.185674 21193 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 12 to 13, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:20.191556 23231 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:20.192034 21729 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:20.192284 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:20.192584 29364 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20250110 03:07:20.192720 23167 consensus_peers.cc:487] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:20.192723 29364 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:20.193759 29714 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:20.193863 21729 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:20.194833 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:20.194893 21188 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 13 to 14, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 2 leader_uuid: "b2e7bce763e24b63bc760b0b82b53454" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:20.206622 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.208453 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:20.211612 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:20.218531 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.218694 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.218784 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:20.219862 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:20.222615 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.239773 22968 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:20.240394 23102 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:20.240422 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
W20250110 03:07:20.240661 22904 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:20.240723 29927 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.240864 29928 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.241712 29928 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:20.241797 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:20.241803 23102 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:20.243026 21188 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 15 to 16, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:20.283331 29609 raft_consensus.cc:1059] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:20.283505 29609 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:20.283965 21729 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:20.283968 23102 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:20.283979 23231 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:20.284174 28023 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:20.284304 29605 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:20.284476 28024 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:20.285024 29605 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.285130 23231 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.286341 23102 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.286470 21193 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.286897 21729 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.288060 23364 consensus_queue.cc:237] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:20.288412 23102 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:20.288406 21729 raft_consensus.cc:1270] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:20.288573 28024 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.288645 28023 consensus_queue.cc:1035] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.289218 29605 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.289345 23102 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.289389 21729 raft_consensus.cc:2949] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.290413 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ea70c2cc77384698aedb06bee2b639ce with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.290483 21193 catalog_manager.cc:5526] T ea70c2cc77384698aedb06bee2b639ce P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 15 to 16, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.291366 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet ea70c2cc77384698aedb06bee2b639ce with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:20.291587 29940 tablet_replica.cc:331] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:20.291647 29940 raft_consensus.cc:2238] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.291750 29940 raft_consensus.cc:2267] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.291996 29940 ts_tablet_manager.cc:1905] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.293326 29940 ts_tablet_manager.cc:1918] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250110 03:07:20.293392 29940 log.cc:1198] T ea70c2cc77384698aedb06bee2b639ce P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:20.293684 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet ea70c2cc77384698aedb06bee2b639ce (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:20.297233 21729 raft_consensus.cc:1212] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.19->[2.20-2.20] Dedup: 2.20->[]
I20250110 03:07:20.318948 22968 raft_consensus.cc:1212] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20250110 03:07:20.344101 28767 raft_consensus.cc:1059] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:20.344283 28767 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:20.344758 21337 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:20.344797 23102 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:20.344878 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:20.345003 29968 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:20.345113 28767 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:20.345225 29967 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:20.346092 29968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.346266 23102 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.347106 21337 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.347340 21188 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 16 to 17, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.348547 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.348618 21468 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:20.349022 21337 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:20.349059 23102 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20250110 03:07:20.349239 29967 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:20.349318 29968 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:20.350034 29967 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.350195 23102 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.350198 21337 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.351040 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.351387 21188 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 17 to 18, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.352063 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet b5064ab85103488f873fffef55fd78fd with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:20.352267 29972 tablet_replica.cc:331] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:20.352336 29972 raft_consensus.cc:2238] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.352463 29972 raft_consensus.cc:2267] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.352650 29972 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.353822 29972 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.17
I20250110 03:07:20.353873 29972 log.cc:1198] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/b5064ab85103488f873fffef55fd78fd
I20250110 03:07:20.354216 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet b5064ab85103488f873fffef55fd78fd (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:20.387876 21337 raft_consensus.cc:1212] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:20.403831 21598 raft_consensus.cc:1212] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:20.421193 27286 raft_consensus.cc:1059] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:20.421393 27286 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:20.421871 21598 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:20.421955 21337 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:20.421950 23364 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:20.421950 22968 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:20.422173 29979 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:20.422253 29778 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:20.422355 29599 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:20.422477 28811 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:20.423199 29599 raft_consensus.cc:1022] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: attempt to promote peer c57c9dfa476f4d21b79f81114c5d74cb: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250110 03:07:20.423264 29599 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:20.423429 21337 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:20.424556 22968 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:20.424665 21598 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:20.424845 21193 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.425897 29778 raft_consensus.cc:1059] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951: attempting to promote NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb to VOTER
I20250110 03:07:20.425791 23364 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:20.426092 29778 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.426606 21337 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.427228 21598 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.427425 23364 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.427623 29599 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.427713 28811 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.427850 28811 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.428040 22968 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.428258 29599 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.428728 29599 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.429435 21337 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.429602 21598 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.430047 23364 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.430368 22968 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.430902 21192 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.432140 21729 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.432549 21598 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:20.432581 21337 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:20.432734 23364 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:20.432763 29979 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.432910 29979 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.432982 28811 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.433734 28811 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.434721 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9b16001c451b455f9eda42f3c98a2fdd with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.434856 21337 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.435242 23364 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.435839 21191 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.436416 21598 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.437538 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b16001c451b455f9eda42f3c98a2fdd with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:20.437656 29972 tablet_replica.cc:331] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:20.437723 29972 raft_consensus.cc:2238] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.437808 29972 raft_consensus.cc:2267] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.438117 29972 ts_tablet_manager.cc:1905] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.438119 21729 consensus_queue.cc:237] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.439615 29972 ts_tablet_manager.cc:1918] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:20.439695 29972 log.cc:1198] T 9b16001c451b455f9eda42f3c98a2fdd P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:20.440101 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 9b16001c451b455f9eda42f3c98a2fdd (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:20.440241 23364 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:20.440241 21598 raft_consensus.cc:1270] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer 9da9d601b7d841b5a8f2c2c53f62a951: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:20.440443 27286 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.440629 29778 consensus_queue.cc:1035] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.441248 29778 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.442147 21183 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9b16001c451b455f9eda42f3c98a2fdd with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.442337 23364 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.442693 21191 catalog_manager.cc:5526] T 9b16001c451b455f9eda42f3c98a2fdd P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:20.443732 21598 raft_consensus.cc:2949] T 9b16001c451b455f9eda42f3c98a2fdd P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.444046 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b16001c451b455f9eda42f3c98a2fdd with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:20.444136 29719 tablet_replica.cc:331] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:20.444572 29719 raft_consensus.cc:2238] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.444720 29719 raft_consensus.cc:2267] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.445261 29719 ts_tablet_manager.cc:1905] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.447108 29719 ts_tablet_manager.cc:1918] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:20.447165 29719 log.cc:1198] T 9b16001c451b455f9eda42f3c98a2fdd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:20.447489 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 9b16001c451b455f9eda42f3c98a2fdd (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:20.455099 21598 raft_consensus.cc:1212] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20250110 03:07:20.483891 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.486395 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:20.486924 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.488655 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:20.492559 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:20.495400 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.495886 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:20.496796 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.537528 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:20.542980 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.543386 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.545246 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:20.546605 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:20.554019 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.558698 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.559338 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:20.601675 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:20.603559 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:20.608248 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.608809 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.609300 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:20.611696 30161 ts_tablet_manager.cc:927] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:20.611878 30161 tablet_copy_client.cc:323] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:20.612128 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 41db68f089644324af518422d2cad2fc from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:20.612196 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 41db68f089644324af518422d2cad2fc from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-41db68f089644324af518422d2cad2fc
I20250110 03:07:20.612707 21620 tablet_copy_source_session.cc:215] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:20.613197 30161 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:20.614931 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.615146 30161 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.19
I20250110 03:07:20.615231 30161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db68f089644324af518422d2cad2fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:20.615854 30161 tablet_copy_client.cc:806] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:20.615942 30161 tablet_copy_client.cc:670] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:20.616571 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.620047 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:20.620057 30161 tablet_copy_client.cc:538] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:20.620795 30161 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:20.626139 30161 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=21 overwritten=0 applied=21 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:20.626353 30161 tablet_bootstrap.cc:492] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:20.626446 30161 ts_tablet_manager.cc:1397] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250110 03:07:20.626579 30161 raft_consensus.cc:357] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:20.626668 30161 raft_consensus.cc:738] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:20.626708 30161 consensus_queue.cc:260] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 1.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:20.626824 30161 ts_tablet_manager.cc:1428] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.004s sys 0.000s
I20250110 03:07:20.626996 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-41db68f089644324af518422d2cad2fc received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:20.627055 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-41db68f089644324af518422d2cad2fc on tablet 41db68f089644324af518422d2cad2fc with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:20.661121 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:20.662525 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.666786 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:20.668164 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.670117 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:20.671808 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:20.680464 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.680495 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:20.698323 30223 ts_tablet_manager.cc:927] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:20.698453 30223 tablet_copy_client.cc:323] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.133:42091
I20250110 03:07:20.698709 22991 tablet_copy_service.cc:140] P e4d83809ce4746d5ad65e8462857fd9c: Received BeginTabletCopySession request for tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:42983)
I20250110 03:07:20.698776 22991 tablet_copy_service.cc:161] P e4d83809ce4746d5ad65e8462857fd9c: Beginning new tablet copy session on tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:42983: session id = d0be0af85dee41dd8d01ef4b2af7726f-81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:20.699303 22991 tablet_copy_source_session.cc:215] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:20.699663 30223 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:20.701088 30223 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.12
I20250110 03:07:20.701180 30223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:20.701754 30223 tablet_copy_client.cc:806] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:20.701853 30223 tablet_copy_client.cc:670] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:20.702502 30223 tablet_copy_client.cc:538] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:20.702996 30223 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:20.706173 30223 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:20.706334 30223 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:20.706403 30223 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:20.706481 30223 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:20.706538 30223 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:20.706651 30223 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:20.706745 30223 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s
I20250110 03:07:20.706930 22991 tablet_copy_service.cc:342] P e4d83809ce4746d5ad65e8462857fd9c: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-81b3bd116ad54a9fa10d2d0c829bb467 received from {username='slave'} at 127.14.19.136:42983
I20250110 03:07:20.706990 22991 tablet_copy_service.cc:434] P e4d83809ce4746d5ad65e8462857fd9c: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-81b3bd116ad54a9fa10d2d0c829bb467 on tablet 81b3bd116ad54a9fa10d2d0c829bb467 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:20.742924 29853 raft_consensus.cc:1059] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:20.743163 29853 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:20.743754 21598 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:20.743758 22968 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:20.743978 30227 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.744149 29854 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.744191 21729 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20250110 03:07:20.744622 29854 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:20.744931 29861 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.746008 22968 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.746320 21729 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.748818 21190 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 15 to 16, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.749056 21598 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.750245 21337 consensus_queue.cc:237] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:20.750614 21729 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:20.750633 22968 raft_consensus.cc:1270] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20250110 03:07:20.750792 29854 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:20.750885 29853 consensus_queue.cc:1035] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:20.751356 29854 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.751466 22968 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.751571 21729 raft_consensus.cc:2949] T 7dbef1612c8b4e2baa670b1cbe0e8960 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.752571 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.752590 21194 catalog_manager.cc:5526] T 7dbef1612c8b4e2baa670b1cbe0e8960 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 16 to 17, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:20.753331 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbef1612c8b4e2baa670b1cbe0e8960 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:20.753515 30231 tablet_replica.cc:331] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:20.753579 30231 raft_consensus.cc:2238] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.753666 30231 raft_consensus.cc:2267] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.753893 30231 ts_tablet_manager.cc:1905] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.754958 30231 ts_tablet_manager.cc:1918] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.16
I20250110 03:07:20.755046 30231 log.cc:1198] T 7dbef1612c8b4e2baa670b1cbe0e8960 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:20.755357 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 7dbef1612c8b4e2baa670b1cbe0e8960 (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:20.760944 30232 ts_tablet_manager.cc:927] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887)
I20250110 03:07:20.761059 30232 tablet_copy_client.cc:323] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.135:39887
I20250110 03:07:20.761286 23253 tablet_copy_service.cc:140] P b2e7bce763e24b63bc760b0b82b53454: Received BeginTabletCopySession request for tablet 62aacc8deebd43debd76a2c10815da2f from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:57051)
I20250110 03:07:20.761345 23253 tablet_copy_service.cc:161] P b2e7bce763e24b63bc760b0b82b53454: Beginning new tablet copy session on tablet 62aacc8deebd43debd76a2c10815da2f from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:57051: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:20.761788 23253 tablet_copy_source_session.cc:215] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:20.762099 30232 ts_tablet_manager.cc:1905] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:20.763479 30232 ts_tablet_manager.cc:1918] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.12
I20250110 03:07:20.763556 30232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62aacc8deebd43debd76a2c10815da2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:20.764045 30232 tablet_copy_client.cc:806] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:20.764161 30232 tablet_copy_client.cc:670] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:20.764760 30232 tablet_copy_client.cc:538] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:20.765237 30232 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:20.768534 30232 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:20.768705 30232 tablet_bootstrap.cc:492] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:20.768788 30232 ts_tablet_manager.cc:1397] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250110 03:07:20.768887 30232 raft_consensus.cc:357] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:20.768946 30232 raft_consensus.cc:738] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:20.768988 30232 consensus_queue.cc:260] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:20.769093 30232 ts_tablet_manager.cc:1428] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:20.769357 23253 tablet_copy_service.cc:342] P b2e7bce763e24b63bc760b0b82b53454: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-62aacc8deebd43debd76a2c10815da2f received from {username='slave'} at 127.14.19.129:57051
I20250110 03:07:20.769418 23253 tablet_copy_service.cc:434] P b2e7bce763e24b63bc760b0b82b53454: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-62aacc8deebd43debd76a2c10815da2f on tablet 62aacc8deebd43debd76a2c10815da2f with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:20.770020 28846 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:20.770193 28846 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 2.20, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:20.770709 21729 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:20.770754 21468 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:20.770886 30236 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:20.770960 23231 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:20.770982 28680 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:20.771127 28846 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:20.771869 28680 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.21: config changed from index 20 to 21, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.772080 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.772722 21468 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.772856 21729 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.773818 21189 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 20 to 21, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.774286 30223 ts_tablet_manager.cc:927] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257)
I20250110 03:07:20.774451 30223 tablet_copy_client.cc:323] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:45257
I20250110 03:07:20.774757 21358 tablet_copy_service.cc:140] P a2b4fdb2c41d4a7ebf55cfac417c3422: Received BeginTabletCopySession request for tablet 58e33e63f48f4b758c19774888345ee0 from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:57367)
I20250110 03:07:20.774808 21358 tablet_copy_service.cc:161] P a2b4fdb2c41d4a7ebf55cfac417c3422: Beginning new tablet copy session on tablet 58e33e63f48f4b758c19774888345ee0 from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:57367: session id = d0be0af85dee41dd8d01ef4b2af7726f-58e33e63f48f4b758c19774888345ee0
I20250110 03:07:20.774804 21598 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 2.21, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:20.775308 21358 tablet_copy_source_session.cc:215] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:20.775611 30223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e33e63f48f4b758c19774888345ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:20.775784 21729 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 21. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20250110 03:07:20.775799 21468 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 21. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20250110 03:07:20.775961 28843 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:20.776077 28843 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:20.776634 30236 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.22: config changed from index 21 to 22, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.776736 21729 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.22: config changed from index 21 to 22, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.777163 30223 tablet_copy_client.cc:806] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:20.777860 30223 tablet_copy_client.cc:670] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:20.777797 21194 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 21 to 22, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.778147 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 21: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.776784 21468 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.22: config changed from index 21 to 22, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:20.778417 30223 tablet_copy_client.cc:538] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:20.779019 30223 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:20.779947 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 22) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:20.780042 29940 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:20.780104 29940 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.780228 29940 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.782671 29940 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.783926 29940 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.21
I20250110 03:07:20.783984 29940 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:20.784320 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:20.784734 30223 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:20.784982 30223 tablet_bootstrap.cc:492] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:20.785084 30223 ts_tablet_manager.cc:1397] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250110 03:07:20.785197 30223 raft_consensus.cc:357] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:20.785249 30223 raft_consensus.cc:738] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:20.785279 30223 consensus_queue.cc:260] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:20.785362 30223 ts_tablet_manager.cc:1428] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:20.785609 21358 tablet_copy_service.cc:342] P a2b4fdb2c41d4a7ebf55cfac417c3422: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-58e33e63f48f4b758c19774888345ee0 received from {username='slave'} at 127.14.19.136:57367
I20250110 03:07:20.785672 21358 tablet_copy_service.cc:434] P a2b4fdb2c41d4a7ebf55cfac417c3422: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-58e33e63f48f4b758c19774888345ee0 on tablet 58e33e63f48f4b758c19774888345ee0 with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:20.799870 28846 raft_consensus.cc:1059] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:20.800032 28846 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:20.800484 23364 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:20.800558 21468 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:20.800760 30236 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:20.800849 28843 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:20.800901 21337 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:20.801457 28843 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:20.801649 28680 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.801797 23364 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.802333 21337 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.802780 21468 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.802989 21194 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 7 to 8, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.804188 21598 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:20.804752 23364 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:20.805589 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:20.805769 28846 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:20.805863 28843 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:20.806856 23364 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.806949 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.807713 28846 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.808106 21190 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 8 to 9, VOTER ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:20.808751 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c5d0922741634918830ee2c163cbce24 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.809191 21447 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d0922741634918830ee2c163cbce24 with delete_type TABLET_DATA_TOMBSTONED (TS ab3ed3d8c2394f78943461e22853c7ac not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52398
I20250110 03:07:20.809352 30281 tablet_replica.cc:331] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: stopping tablet replica
I20250110 03:07:20.809420 30281 raft_consensus.cc:2238] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.809516 30281 raft_consensus.cc:2267] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.809760 30281 ts_tablet_manager.cc:1905] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.810971 30281 ts_tablet_manager.cc:1918] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:20.811023 30281 log.cc:1198] T c5d0922741634918830ee2c163cbce24 P ab3ed3d8c2394f78943461e22853c7ac: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/c5d0922741634918830ee2c163cbce24
I20250110 03:07:20.811434 21182 catalog_manager.cc:4872] TS ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469): tablet c5d0922741634918830ee2c163cbce24 (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:20.988173 30227 raft_consensus.cc:1059] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb to VOTER
I20250110 03:07:20.988291 30317 raft_consensus.cc:1059] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb to VOTER
I20250110 03:07:20.988399 30227 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.988523 30317 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.989032 23229 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:20.989151 22968 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:20.989171 21598 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:20.989346 29853 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.989387 23102 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.989439 21335 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.989456 21598 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:20.989578 29861 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.989681 29861 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:20.990001 28767 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.990164 28767 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.990267 28767 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:20.990561 29854 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.990713 21599 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.991461 30320 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.991955 21194 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 10 to 11, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.992249 22968 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.992697 21335 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.992717 23102 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.993557 21190 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:20.994041 23229 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.994272 21335 consensus_queue.cc:237] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.995110 23231 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:20.995396 30227 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.995487 21598 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.995733 21599 raft_consensus.cc:1270] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:20.995890 30227 consensus_queue.cc:1035] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:20.996495 29861 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.997277 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5a1a954889a9452f8a8ee55aa0a6f26e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:20.997269 21468 consensus_queue.cc:237] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:20.997299 21598 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.997975 21188 catalog_manager.cc:5526] T 5a1a954889a9452f8a8ee55aa0a6f26e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:20.998781 23231 raft_consensus.cc:2949] T 5a1a954889a9452f8a8ee55aa0a6f26e P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:20.998837 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a1a954889a9452f8a8ee55aa0a6f26e with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:20.999042 30329 tablet_replica.cc:331] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:20.999220 30329 raft_consensus.cc:2238] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:20.999344 30329 raft_consensus.cc:2267] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:20.999603 30329 ts_tablet_manager.cc:1905] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:20.999838 21335 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:21.000065 21598 raft_consensus.cc:1270] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:21.000065 30317 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:21.000324 30317 consensus_queue.cc:1035] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:21.000645 28767 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:21.000903 21335 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:21.001466 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 87f8afb021a7407089c710f6dd588d03 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:21.001740 21194 catalog_manager.cc:5526] T 87f8afb021a7407089c710f6dd588d03 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 10 to 11, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:21.001986 21598 raft_consensus.cc:2949] T 87f8afb021a7407089c710f6dd588d03 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 531cf07bcce6435c9e18185c60b2118d (127.14.19.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:21.002128 30329 ts_tablet_manager.cc:1918] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:21.002182 30329 log.cc:1198] T 5a1a954889a9452f8a8ee55aa0a6f26e P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:21.002521 23082 tablet_service.cc:1514] Processing DeleteTablet for tablet 87f8afb021a7407089c710f6dd588d03 with delete_type TABLET_DATA_TOMBSTONED (TS 531cf07bcce6435c9e18185c60b2118d not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:52468
I20250110 03:07:21.002849 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 5a1a954889a9452f8a8ee55aa0a6f26e (table rebalance_test_table_5 [id=71234a92f0d94d8f919e65f5a2f37654]) successfully deleted
I20250110 03:07:21.002907 30330 tablet_replica.cc:331] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: stopping tablet replica
I20250110 03:07:21.002993 30330 raft_consensus.cc:2238] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:21.003082 30330 raft_consensus.cc:2267] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:21.003302 30330 ts_tablet_manager.cc:1905] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:21.004454 30330 ts_tablet_manager.cc:1918] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:21.004501 30330 log.cc:1198] T 87f8afb021a7407089c710f6dd588d03 P 531cf07bcce6435c9e18185c60b2118d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/87f8afb021a7407089c710f6dd588d03
I20250110 03:07:21.004822 21181 catalog_manager.cc:4872] TS 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351): tablet 87f8afb021a7407089c710f6dd588d03 (table rebalance_test_table_3 [id=5983e55a26094cf695eb945bafac240d]) successfully deleted
I20250110 03:07:21.059165 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.059379 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:21.059541 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.061542 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:21.068022 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.071251 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:21.076247 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.076522 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:21.088821 23364 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 2.21, Last appended by leader: 18, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:21.089409 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 21. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
W20250110 03:07:21.089442 23298 consensus_peers.cc:487] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:21.089540 21468 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 21. Preceding OpId from leader: term: 2 index: 22. (index mismatch)
I20250110 03:07:21.089593 30390 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:21.089766 30390 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:21.090410 30389 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.22: config changed from index 21 to 22, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:21.090564 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.22: config changed from index 21 to 22, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:21.091725 21188 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 21 to 22, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:21.092414 21468 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.22: config changed from index 21 to 22, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:21.096920 21598 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:21.097420 23364 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:21.097604 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:21.097646 28680 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20250110 03:07:21.097741 21534 consensus_peers.cc:487] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5d0922741634918830ee2c163cbce24. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:21.097940 28680 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:21.098613 28680 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:21.098819 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:21.100081 23364 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:21.100287 21188 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 9 to 10, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:21.187327 23364 raft_consensus.cc:1212] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20250110 03:07:21.195722 22968 raft_consensus.cc:1212] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.20->[1.21-1.21] Dedup: 1.21->[]
I20250110 03:07:21.268774 21335 raft_consensus.cc:1212] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20250110 03:07:21.279136 23364 raft_consensus.cc:1212] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.15->[2.16-2.16] Dedup: 2.16->[]
I20250110 03:07:21.555274 30487 ts_tablet_manager.cc:927] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Initiating tablet copy from peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:21.555420 30487 tablet_copy_client.cc:323] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.136:42591
I20250110 03:07:21.555642 23370 tablet_copy_service.cc:140] P d0be0af85dee41dd8d01ef4b2af7726f: Received BeginTabletCopySession request for tablet 32609d619a0242d1ba0a851fa0a7812d from peer c57c9dfa476f4d21b79f81114c5d74cb ({username='slave'} at 127.14.19.131:44395)
I20250110 03:07:21.555704 23370 tablet_copy_service.cc:161] P d0be0af85dee41dd8d01ef4b2af7726f: Beginning new tablet copy session on tablet 32609d619a0242d1ba0a851fa0a7812d from peer c57c9dfa476f4d21b79f81114c5d74cb at {username='slave'} at 127.14.19.131:44395: session id = c57c9dfa476f4d21b79f81114c5d74cb-32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:21.556296 23370 tablet_copy_source_session.cc:215] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:21.556607 30487 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:21.558193 30487 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.20
I20250110 03:07:21.558297 30487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32609d619a0242d1ba0a851fa0a7812d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:21.558880 30487 tablet_copy_client.cc:806] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:21.558967 30487 tablet_copy_client.cc:670] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:21.559561 30487 tablet_copy_client.cc:538] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:21.560094 30487 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:21.563803 30487 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:21.563980 30487 tablet_bootstrap.cc:492] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap complete.
I20250110 03:07:21.564069 30487 ts_tablet_manager.cc:1397] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250110 03:07:21.564208 30487 raft_consensus.cc:357] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:21.564275 30487 raft_consensus.cc:738] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: LEARNER
I20250110 03:07:21.564299 30487 consensus_queue.cc:260] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 2.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:21.564404 30487 ts_tablet_manager.cc:1428] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:21.564592 23370 tablet_copy_service.cc:342] P d0be0af85dee41dd8d01ef4b2af7726f: Request end of tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-32609d619a0242d1ba0a851fa0a7812d received from {username='slave'} at 127.14.19.131:44395
I20250110 03:07:21.564649 23370 tablet_copy_service.cc:434] P d0be0af85dee41dd8d01ef4b2af7726f: ending tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-32609d619a0242d1ba0a851fa0a7812d on tablet 32609d619a0242d1ba0a851fa0a7812d with peer c57c9dfa476f4d21b79f81114c5d74cb
I20250110 03:07:21.713646 30496 raft_consensus.cc:1059] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:21.713876 30496 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:21.714387 23364 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:21.714390 23231 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:21.714421 23102 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:21.714617 30498 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:21.714738 29853 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:21.714845 30499 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:21.715534 30499 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.715705 23364 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.715715 23231 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.716820 21190 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.717147 23102 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.717576 30504 ts_tablet_manager.cc:927] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:21.717705 30504 tablet_copy_client.cc:323] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:21.718227 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet c5d0922741634918830ee2c163cbce24 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:34311)
I20250110 03:07:21.718287 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet c5d0922741634918830ee2c163cbce24 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:34311: session id = e4d83809ce4746d5ad65e8462857fd9c-c5d0922741634918830ee2c163cbce24
I20250110 03:07:21.718369 21335 consensus_queue.cc:237] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:21.718760 23364 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:21.718799 23102 raft_consensus.cc:1270] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer a2b4fdb2c41d4a7ebf55cfac417c3422: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20250110 03:07:21.718910 21620 tablet_copy_source_session.cc:215] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:21.718930 30498 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:21.719064 30499 consensus_queue.cc:1035] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:21.719321 30504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:21.719674 30499 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.719803 23102 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.720639 23364 raft_consensus.cc:2949] T 58e33e63f48f4b758c19774888345ee0 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.720877 21189 catalog_manager.cc:5526] T 58e33e63f48f4b758c19774888345ee0 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 14 to 15, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.720938 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 58e33e63f48f4b758c19774888345ee0 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:21.721030 30504 tablet_copy_client.cc:806] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:21.721140 30504 tablet_copy_client.cc:670] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:21.721818 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e33e63f48f4b758c19774888345ee0 with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:21.721997 30504 tablet_copy_client.cc:538] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:21.722003 30509 tablet_replica.cc:331] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:21.722064 30509 raft_consensus.cc:2238] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:21.722173 30509 raft_consensus.cc:2267] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:21.722410 30509 ts_tablet_manager.cc:1905] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:21.722651 30504 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:21.723582 30509 ts_tablet_manager.cc:1918] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20250110 03:07:21.723628 30509 log.cc:1198] T 58e33e63f48f4b758c19774888345ee0 P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/58e33e63f48f4b758c19774888345ee0
I20250110 03:07:21.723956 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 58e33e63f48f4b758c19774888345ee0 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:21.725862 30504 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:21.726027 30504 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:21.726112 30504 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:21.726215 30504 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:21.726276 30504 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:21.726310 30504 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:21.726411 30504 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:21.726660 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-c5d0922741634918830ee2c163cbce24 received from {username='slave'} at 127.14.19.133:34311
I20250110 03:07:21.726727 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-c5d0922741634918830ee2c163cbce24 on tablet c5d0922741634918830ee2c163cbce24 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:21.729620 30425 raft_consensus.cc:1059] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:21.729831 30425 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:21.730371 21729 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:21.730623 29717 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:21.731451 22968 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:21.731381 21335 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:21.731753 30424 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:21.731879 30424 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:21.733116 30424 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.734171 21729 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.735286 21190 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 reported cstate change: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b2e7bce763e24b63bc760b0b82b53454" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:21.735324 22968 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.736311 21335 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.737434 23231 consensus_queue.cc:237] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:21.737919 21335 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:21.737927 21729 raft_consensus.cc:1270] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer b2e7bce763e24b63bc760b0b82b53454: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:21.738097 30424 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:21.738198 30521 consensus_queue.cc:1035] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:21.739135 29717 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.739239 21729 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.739320 21335 raft_consensus.cc:2949] T 62aacc8deebd43debd76a2c10815da2f P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.740092 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 62aacc8deebd43debd76a2c10815da2f with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:21.740420 21188 catalog_manager.cc:5526] T 62aacc8deebd43debd76a2c10815da2f P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 2 leader_uuid: "b2e7bce763e24b63bc760b0b82b53454" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:21.741515 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 62aacc8deebd43debd76a2c10815da2f with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:21.742174 30536 tablet_replica.cc:331] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:21.742244 30536 raft_consensus.cc:2238] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:21.742347 30536 raft_consensus.cc:2267] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:21.742599 30536 ts_tablet_manager.cc:1905] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:21.744261 30536 ts_tablet_manager.cc:1918] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250110 03:07:21.744310 30536 log.cc:1198] T 62aacc8deebd43debd76a2c10815da2f P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:21.744678 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 62aacc8deebd43debd76a2c10815da2f (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:21.767014 30557 raft_consensus.cc:1059] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:21.767201 30557 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:21.767704 23231 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250110 03:07:21.767737 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250110 03:07:21.767848 21335 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20250110 03:07:21.767936 30560 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:21.768062 30559 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:21.768261 30557 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:21.768803 30236 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.769129 23231 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.769804 21335 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.769869 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.770856 21190 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 21 to 22, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:21.771782 21598 consensus_queue.cc:237] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:21.772154 22968 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250110 03:07:21.772215 23231 raft_consensus.cc:1270] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20250110 03:07:21.772310 30557 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:21.772403 30559 consensus_queue.cc:1035] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:21.772907 30236 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.773047 23231 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.773097 22968 raft_consensus.cc:2949] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.773885 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 41db68f089644324af518422d2cad2fc with cas_config_opid_index 22: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:21.774196 21190 catalog_manager.cc:5526] T 41db68f089644324af518422d2cad2fc P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 22 to 23, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:21.774832 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet 41db68f089644324af518422d2cad2fc with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 23) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:21.775025 30564 tablet_replica.cc:331] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:21.775081 30564 raft_consensus.cc:2238] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:21.775123 30564 raft_consensus.cc:2267] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:21.775339 30564 ts_tablet_manager.cc:1905] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:21.776480 30564 ts_tablet_manager.cc:1918] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.22
I20250110 03:07:21.776533 30564 log.cc:1198] T 41db68f089644324af518422d2cad2fc P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/41db68f089644324af518422d2cad2fc
I20250110 03:07:21.776823 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet 41db68f089644324af518422d2cad2fc (table rebalance_test_table_1 [id=44b4dd94d5944b82a62d2e8bc2df75fa]) successfully deleted
I20250110 03:07:21.784479 30431 raft_consensus.cc:1059] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:21.784631 30431 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:21.785038 23364 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEARNER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250110 03:07:21.785157 23102 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250110 03:07:21.785161 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250110 03:07:21.785224 30567 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:21.785400 30565 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:21.785513 30568 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:21.786252 30568 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.17: config changed from index 16 to 17, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.786384 23102 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.786478 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.787421 23364 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:21.787537 21194 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 16 to 17, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:21.895442 22968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44592
I20250110 03:07:21.895540 22968 raft_consensus.cc:604] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Received request to transfer leadership
I20250110 03:07:21.987840 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:21.987712 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.995149 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:21.995307 23344 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.995545 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:21.995858 21580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20250110 03:07:21.996299 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:21.999415 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:22.018621 21468 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:22.019162 21335 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250110 03:07:22.019188 23102 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20250110 03:07:22.019407 28767 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20250110 03:07:22.019542 21403 consensus_peers.cc:487] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:22.019644 30490 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:22.020231 30321 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:22.020386 21335 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:22.021657 21195 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 18 to 19, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:22.021833 23102 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:22.025988 21598 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 2.22, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:22.026448 21468 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250110 03:07:22.026475 21729 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
W20250110 03:07:22.026623 21535 consensus_peers.cc:487] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb -> Peer b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): Couldn't send request to peer b2e7bce763e24b63bc760b0b82b53454. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:22.026715 30236 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:22.026820 30559 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:22.027469 30557 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.23: config changed from index 22 to 23, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:22.027693 21729 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:22.027715 21468 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:22.028705 21188 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 reported cstate change: config changed from index 22 to 23, NON_VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) added. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } } }
I20250110 03:07:22.032812 22968 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:22.033417 23102 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20250110 03:07:22.033416 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
W20250110 03:07:22.033623 22905 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c -> Peer a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): Couldn't send request to peer a2b4fdb2c41d4a7ebf55cfac417c3422. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:22.033627 30565 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:22.033768 30567 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250110 03:07:22.034421 30565 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:22.034597 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:22.034605 23102 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:22.035658 21196 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 16 to 17, NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) added. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } }
I20250110 03:07:22.052621 21598 raft_consensus.cc:1212] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.21->[2.22-2.22] Dedup: 2.22->[]
I20250110 03:07:22.102653 22968 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44640
I20250110 03:07:22.102736 22968 raft_consensus.cc:604] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Received request to transfer leadership
I20250110 03:07:22.168054 30567 raft_consensus.cc:988] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: : Instructing follower d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:22.168123 30567 raft_consensus.cc:1076] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Signalling peer d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:22.168323 23364 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
from {username='slave'} at 127.14.19.133:35405
I20250110 03:07:22.168404 23364 raft_consensus.cc:491] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:22.168434 23364 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:22.168996 23364 raft_consensus.cc:513] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:22.169174 23364 leader_election.cc:290] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 3 election: Requested vote from peers 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355), 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351), e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:22.169313 21729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:22.169361 22968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: true dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
I20250110 03:07:22.169412 21729 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:22.169432 22968 raft_consensus.cc:3049] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Stepping down as leader of term 2
I20250110 03:07:22.169379 23102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: true dest_uuid: "531cf07bcce6435c9e18185c60b2118d"
I20250110 03:07:22.169463 22968 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Running, Role: LEADER
I20250110 03:07:22.169472 23102 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:22.169517 22968 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:22.169607 22968 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Advancing to term 3
I20250110 03:07:22.170138 23102 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 3.
I20250110 03:07:22.170215 21729 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 3.
I20250110 03:07:22.170339 22968 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 3.
I20250110 03:07:22.170403 23300 leader_election.cc:304] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 531cf07bcce6435c9e18185c60b2118d, 9da9d601b7d841b5a8f2c2c53f62a951, d0be0af85dee41dd8d01ef4b2af7726f; no voters:
I20250110 03:07:22.170507 28024 raft_consensus.cc:2798] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 FOLLOWER]: Leader election won for term 3
I20250110 03:07:22.170603 28024 raft_consensus.cc:695] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Becoming Leader. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Running, Role: LEADER
I20250110 03:07:22.170668 28024 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:22.171190 21196 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: term changed from 2 to 3, leader changed from e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) to d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136). New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:22.239575 22968 raft_consensus.cc:1212] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20250110 03:07:22.512566 30750 ts_tablet_manager.cc:927] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Initiating tablet copy from peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:22.512697 30750 tablet_copy_client.cc:323] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.133:42091
I20250110 03:07:22.512920 22991 tablet_copy_service.cc:140] P e4d83809ce4746d5ad65e8462857fd9c: Received BeginTabletCopySession request for tablet 250f547c54a84a35b2850d50b897810e from peer a2b4fdb2c41d4a7ebf55cfac417c3422 ({username='slave'} at 127.14.19.129:48197)
I20250110 03:07:22.512986 22991 tablet_copy_service.cc:161] P e4d83809ce4746d5ad65e8462857fd9c: Beginning new tablet copy session on tablet 250f547c54a84a35b2850d50b897810e from peer a2b4fdb2c41d4a7ebf55cfac417c3422 at {username='slave'} at 127.14.19.129:48197: session id = a2b4fdb2c41d4a7ebf55cfac417c3422-250f547c54a84a35b2850d50b897810e
I20250110 03:07:22.513473 22991 tablet_copy_source_session.cc:215] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:22.513739 30750 ts_tablet_manager.cc:1905] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:22.515050 30750 ts_tablet_manager.cc:1918] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.15
I20250110 03:07:22.515128 30750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250f547c54a84a35b2850d50b897810e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:22.515597 30750 tablet_copy_client.cc:806] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:22.515689 30750 tablet_copy_client.cc:670] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:22.516263 30750 tablet_copy_client.cc:538] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:22.516749 30750 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap starting.
I20250110 03:07:22.519953 30750 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:22.520139 30750 tablet_bootstrap.cc:492] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Bootstrap complete.
I20250110 03:07:22.520227 30750 ts_tablet_manager.cc:1397] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:22.520359 30750 raft_consensus.cc:357] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:22.520426 30750 raft_consensus.cc:738] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a2b4fdb2c41d4a7ebf55cfac417c3422, State: Initialized, Role: LEARNER
I20250110 03:07:22.520452 30750 consensus_queue.cc:260] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 2.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } }
I20250110 03:07:22.520550 30750 ts_tablet_manager.cc:1428] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:22.520758 22991 tablet_copy_service.cc:342] P e4d83809ce4746d5ad65e8462857fd9c: Request end of tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-250f547c54a84a35b2850d50b897810e received from {username='slave'} at 127.14.19.129:48197
I20250110 03:07:22.520818 22991 tablet_copy_service.cc:434] P e4d83809ce4746d5ad65e8462857fd9c: ending tablet copy session a2b4fdb2c41d4a7ebf55cfac417c3422-250f547c54a84a35b2850d50b897810e on tablet 250f547c54a84a35b2850d50b897810e with peer a2b4fdb2c41d4a7ebf55cfac417c3422
I20250110 03:07:22.555045 30753 ts_tablet_manager.cc:927] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Initiating tablet copy from peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011)
I20250110 03:07:22.555176 30753 tablet_copy_client.cc:323] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:38011
I20250110 03:07:22.555418 21620 tablet_copy_service.cc:140] P c57c9dfa476f4d21b79f81114c5d74cb: Received BeginTabletCopySession request for tablet 5cc5781db40c4148ba1889c45405efc2 from peer b2e7bce763e24b63bc760b0b82b53454 ({username='slave'} at 127.14.19.135:42823)
I20250110 03:07:22.555478 21620 tablet_copy_service.cc:161] P c57c9dfa476f4d21b79f81114c5d74cb: Beginning new tablet copy session on tablet 5cc5781db40c4148ba1889c45405efc2 from peer b2e7bce763e24b63bc760b0b82b53454 at {username='slave'} at 127.14.19.135:42823: session id = b2e7bce763e24b63bc760b0b82b53454-5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:22.555888 21620 tablet_copy_source_session.cc:215] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:22.556146 30753 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:22.557655 30753 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.21
I20250110 03:07:22.557770 30753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cc5781db40c4148ba1889c45405efc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:22.558272 30753 tablet_copy_client.cc:806] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:22.558388 30753 tablet_copy_client.cc:670] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:22.559038 30753 tablet_copy_client.cc:538] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:22.559554 30753 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap starting.
I20250110 03:07:22.563336 30753 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=23 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:22.563529 30753 tablet_bootstrap.cc:492] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Bootstrap complete.
I20250110 03:07:22.563620 30753 ts_tablet_manager.cc:1397] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20250110 03:07:22.563733 30753 raft_consensus.cc:357] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:22.563814 30753 raft_consensus.cc:738] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b2e7bce763e24b63bc760b0b82b53454, State: Initialized, Role: LEARNER
I20250110 03:07:22.563846 30753 consensus_queue.cc:260] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 2.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: NON_VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: true } }
I20250110 03:07:22.563939 30753 ts_tablet_manager.cc:1428] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:22.564267 21620 tablet_copy_service.cc:342] P c57c9dfa476f4d21b79f81114c5d74cb: Request end of tablet copy session b2e7bce763e24b63bc760b0b82b53454-5cc5781db40c4148ba1889c45405efc2 received from {username='slave'} at 127.14.19.135:42823
I20250110 03:07:22.564329 21620 tablet_copy_service.cc:434] P c57c9dfa476f4d21b79f81114c5d74cb: ending tablet copy session b2e7bce763e24b63bc760b0b82b53454-5cc5781db40c4148ba1889c45405efc2 on tablet 5cc5781db40c4148ba1889c45405efc2 with peer b2e7bce763e24b63bc760b0b82b53454
I20250110 03:07:22.582213 22968 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20250110 03:07:22.582437 28024 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:22.623101 30678 raft_consensus.cc:1059] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb to VOTER
I20250110 03:07:22.623299 30678 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 2.22, Last appended by leader: 18, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:22.623824 21729 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250110 03:07:22.623824 21598 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250110 03:07:22.623836 21468 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 22. Preceding OpId from leader: term: 2 index: 23. (index mismatch)
I20250110 03:07:22.624011 28024 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:22.624128 30768 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:22.624246 30678 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:22.624961 30678 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.23: config changed from index 22 to 23, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.625146 21468 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.626030 21729 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.626286 21195 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 22 to 23, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:22.626334 21598 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.23: config changed from index 22 to 23, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.627657 23364 consensus_queue.cc:237] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 2.23, Last appended by leader: 18, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:22.628036 21598 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250110 03:07:22.628036 21468 raft_consensus.cc:1270] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250110 03:07:22.628201 30768 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:22.628297 30767 consensus_queue.cc:1035] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:22.628875 30767 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.24: config changed from index 23 to 24, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.629043 21468 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.629771 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 32609d619a0242d1ba0a851fa0a7812d with cas_config_opid_index 23: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:22.629845 21598 raft_consensus.cc:2949] T 32609d619a0242d1ba0a851fa0a7812d P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:22.629988 21188 catalog_manager.cc:5526] T 32609d619a0242d1ba0a851fa0a7812d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 23 to 24, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:22.630851 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 32609d619a0242d1ba0a851fa0a7812d with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 24) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:22.631013 30772 tablet_replica.cc:331] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:22.631064 30772 raft_consensus.cc:2238] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:22.631135 30772 raft_consensus.cc:2267] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:22.631328 30772 ts_tablet_manager.cc:1905] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:22.632421 30772 ts_tablet_manager.cc:1918] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.23
I20250110 03:07:22.632459 30772 log.cc:1198] T 32609d619a0242d1ba0a851fa0a7812d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:22.632786 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 32609d619a0242d1ba0a851fa0a7812d (table rebalance_test_table_6 [id=f19d99171f2649b2b62d31b96d96ee10]) successfully deleted
I20250110 03:07:22.638218 30773 ts_tablet_manager.cc:927] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:22.638381 30773 tablet_copy_client.cc:323] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:22.638615 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet b5064ab85103488f873fffef55fd78fd from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:59673)
I20250110 03:07:22.638664 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet b5064ab85103488f873fffef55fd78fd from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:59673: session id = e4d83809ce4746d5ad65e8462857fd9c-b5064ab85103488f873fffef55fd78fd
I20250110 03:07:22.639041 21488 tablet_copy_source_session.cc:215] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:22.639297 30773 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:22.640547 30773 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.17
I20250110 03:07:22.640632 30773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5064ab85103488f873fffef55fd78fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:22.641096 30773 tablet_copy_client.cc:806] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:22.641185 30773 tablet_copy_client.cc:670] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:22.641738 30773 tablet_copy_client.cc:538] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:22.642210 30773 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:22.645749 30773 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:22.645933 30773 tablet_bootstrap.cc:492] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:22.646021 30773 ts_tablet_manager.cc:1397] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20250110 03:07:22.646126 30773 raft_consensus.cc:357] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:22.646178 30773 raft_consensus.cc:738] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:22.646206 30773 consensus_queue.cc:260] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:22.646306 30773 ts_tablet_manager.cc:1428] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:22.646529 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-b5064ab85103488f873fffef55fd78fd received from {username='slave'} at 127.14.19.133:59673
I20250110 03:07:22.646600 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-b5064ab85103488f873fffef55fd78fd on tablet b5064ab85103488f873fffef55fd78fd with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:22.671604 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20250110 03:07:22.671818 30768 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:22.673271 23102 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 3 index: 18. (index mismatch)
I20250110 03:07:22.673729 30768 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:22.679656 30559 raft_consensus.cc:1059] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:22.679850 30559 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:22.680387 22966 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:22.680398 23364 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:22.680462 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:22.680622 30797 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:22.680720 30236 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:22.680918 30559 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:22.681730 30559 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:22.682716 22966 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:22.682767 23364 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:22.682801 21196 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:22.683115 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:22.774622 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5d0922741634918830ee2c163cbce24"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45654
I20250110 03:07:22.774708 21598 raft_consensus.cc:604] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:22.784024 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5d0922741634918830ee2c163cbce24"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45672
I20250110 03:07:22.784097 21598 raft_consensus.cc:604] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:22.990935 21598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5d0922741634918830ee2c163cbce24"
dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45698
I20250110 03:07:22.991046 21598 raft_consensus.cc:604] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:23.008986 23231 raft_consensus.cc:1212] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.22->[2.23-2.23] Dedup: 2.23->[]
I20250110 03:07:23.012471 21335 raft_consensus.cc:1212] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.16->[2.17-2.17] Dedup: 2.17->[]
I20250110 03:07:23.022681 22966 raft_consensus.cc:1212] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.18->[1.19-1.19] Dedup: 1.19->[]
I20250110 03:07:23.087658 30559 raft_consensus.cc:988] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: : Instructing follower d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:23.087743 30559 raft_consensus.cc:1076] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Signalling peer d0be0af85dee41dd8d01ef4b2af7726f to start an election
I20250110 03:07:23.087925 23364 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5d0922741634918830ee2c163cbce24"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
from {username='slave'} at 127.14.19.131:44395
I20250110 03:07:23.087999 23364 raft_consensus.cc:491] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:23.088021 23364 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:23.088653 23364 raft_consensus.cc:513] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:23.088783 23364 leader_election.cc:290] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Requested vote from peers c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011), a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257), e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:23.088971 21335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422"
I20250110 03:07:23.088997 21598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "c57c9dfa476f4d21b79f81114c5d74cb"
I20250110 03:07:23.089008 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5d0922741634918830ee2c163cbce24" candidate_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
I20250110 03:07:23.089074 21335 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:23.089073 21598 raft_consensus.cc:3049] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:23.089077 22966 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:23.089105 21598 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Running, Role: LEADER
I20250110 03:07:23.089180 21598 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:23.089282 21598 raft_consensus.cc:3054] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:23.089748 22966 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:23.089923 21335 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:23.090001 21598 raft_consensus.cc:2463] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0be0af85dee41dd8d01ef4b2af7726f in term 2.
I20250110 03:07:23.090126 23298 leader_election.cc:304] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: a2b4fdb2c41d4a7ebf55cfac417c3422, d0be0af85dee41dd8d01ef4b2af7726f, e4d83809ce4746d5ad65e8462857fd9c; no voters:
I20250110 03:07:23.090231 28024 raft_consensus.cc:2798] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:23.090288 28024 raft_consensus.cc:695] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Becoming Leader. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Running, Role: LEADER
I20250110 03:07:23.090348 28024 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:23.090879 21195 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: term changed from 1 to 2, leader changed from c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) to d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136). New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:23.250353 23364 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 3.18, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:23.250823 23102 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20250110 03:07:23.250844 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 18. Preceding OpId from leader: term: 3 index: 19. (index mismatch)
I20250110 03:07:23.251044 30890 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:23.251204 30768 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:23.251906 30890 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Committing config change with OpId 3.19: config changed from index 17 to 19, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:23.252066 23102 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Committing config change with OpId 3.19: config changed from index 17 to 19, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:23.252172 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Committing config change with OpId 3.19: config changed from index 17 to 19, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:23.253135 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:23.253327 21195 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 17 to 19, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:23.254189 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 19) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:23.254516 30892 tablet_replica.cc:331] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:23.254609 30892 raft_consensus.cc:2238] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:23.254657 30892 raft_consensus.cc:2267] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:23.254871 30892 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:23.256065 30892 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.18
I20250110 03:07:23.256109 30892 log.cc:1198] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:23.256448 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet 81b3bd116ad54a9fa10d2d0c829bb467 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:23.354809 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:23.356019 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:23.359148 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:23.361045 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:23.365118 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:23.365778 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:23.366196 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:23.369185 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:23.385486 23364 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 3.19, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:23.385957 23102 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 3 index: 20. (index mismatch)
I20250110 03:07:23.385974 21729 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 19. Preceding OpId from leader: term: 3 index: 20. (index mismatch)
W20250110 03:07:23.386015 23298 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f -> Peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): Couldn't send request to peer e4d83809ce4746d5ad65e8462857fd9c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:23.386195 30768 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.386353 28024 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.386893 30890 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Committing config change with OpId 3.20: config changed from index 19 to 20, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:23.387060 23102 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Committing config change with OpId 3.20: config changed from index 19 to 20, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:23.388100 21729 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Committing config change with OpId 3.20: config changed from index 19 to 20, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } } }
I20250110 03:07:23.388279 21188 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 19 to 20, NON_VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) added. New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:23.393399 22968 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 2.17, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:23.393919 21335 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250110 03:07:23.393934 23102 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
I20250110 03:07:23.394002 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 17. Preceding OpId from leader: term: 2 index: 18. (index mismatch)
W20250110 03:07:23.394131 22907 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:23.394222 30967 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:23.394305 30567 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:23.394389 30966 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250110 03:07:23.395053 30567 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.18: config changed from index 17 to 18, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.395196 23102 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.395226 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.18: config changed from index 17 to 18, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.396104 21188 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 17 to 18, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.397859 21335 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Committing config change with OpId 2.18: config changed from index 17 to 18, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.399009 30567 raft_consensus.cc:1059] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c: attempting to promote NON_VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 to VOTER
I20250110 03:07:23.399147 30567 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:23.399494 30567 raft_consensus.cc:1022] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: attempt to promote peer a2b4fdb2c41d4a7ebf55cfac417c3422: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20250110 03:07:23.399945 22907 consensus_peers.cc:487] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:23.400072 23102 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:23.400079 21335 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 LEARNER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:23.400285 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 18. Preceding OpId from leader: term: 2 index: 19. (index mismatch)
I20250110 03:07:23.400274 30861 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:23.400542 30966 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:23.401044 30966 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250110 03:07:23.401603 30967 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.401804 21335 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.401890 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.402851 21192 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.403249 23100 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.19: config changed from index 18 to 19, a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:23.403702 22968 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 13, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:23.404106 21335 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:23.404150 21598 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:23.404309 23100 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 19. Preceding OpId from leader: term: 2 index: 20. (index mismatch)
I20250110 03:07:23.404307 30567 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.404371 30966 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.404608 30567 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.405048 30966 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.405249 21598 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.405979 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 19: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:23.406092 21188 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.406414 21335 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.406505 23100 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.20: config changed from index 19 to 20, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.406930 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 250f547c54a84a35b2850d50b897810e with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 20) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:23.407662 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 250f547c54a84a35b2850d50b897810e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:23.408004 22966 consensus_queue.cc:237] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 2.20, Last appended by leader: 13, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } }
I20250110 03:07:23.408753 21335 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:23.408792 23100 raft_consensus.cc:1270] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 20. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20250110 03:07:23.408969 30982 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:23.409085 30981 consensus_queue.cc:1035] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:23.409719 30981 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.409848 21335 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.410030 23100 raft_consensus.cc:2949] T 250f547c54a84a35b2850d50b897810e P 531cf07bcce6435c9e18185c60b2118d [term 2 FOLLOWER]: Committing config change with OpId 2.21: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } }
I20250110 03:07:23.410912 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 250f547c54a84a35b2850d50b897810e with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:23.411157 21188 catalog_manager.cc:5526] T 250f547c54a84a35b2850d50b897810e P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 20 to 21, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:23.411830 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet 250f547c54a84a35b2850d50b897810e with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 21) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:23.412123 30989 tablet_replica.cc:331] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:23.412228 30989 raft_consensus.cc:2238] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:23.412407 30989 raft_consensus.cc:2267] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:23.412896 30989 ts_tablet_manager.cc:1905] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:23.414196 30989 ts_tablet_manager.cc:1918] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.20
I20250110 03:07:23.414251 30989 log.cc:1198] T 250f547c54a84a35b2850d50b897810e P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/250f547c54a84a35b2850d50b897810e
I20250110 03:07:23.414577 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet 250f547c54a84a35b2850d50b897810e (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:23.516809 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:23.517035 28024 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:23.526468 21598 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:23.526791 28024 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:23.528402 22968 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 03:07:23.528713 30890 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:23.544157 30559 raft_consensus.cc:1059] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb: attempting to promote NON_VOTER b2e7bce763e24b63bc760b0b82b53454 to VOTER
I20250110 03:07:23.544327 30559 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 2.23, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:23.544797 23231 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 LEARNER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250110 03:07:23.544847 21729 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250110 03:07:23.544953 21468 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 23. Preceding OpId from leader: term: 2 index: 24. (index mismatch)
I20250110 03:07:23.545073 30560 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:23.545195 31027 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:23.545292 31026 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:23.546136 31026 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.24: config changed from index 23 to 24, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.546248 21468 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.547336 21192 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac reported cstate change: config changed from index 23 to 24, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.547894 21729 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.555142 23231 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.24: config changed from index 23 to 24, b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.555586 21598 consensus_queue.cc:237] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 2.24, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } }
I20250110 03:07:23.555927 21468 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20250110 03:07:23.556106 23229 raft_consensus.cc:1270] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer c57c9dfa476f4d21b79f81114c5d74cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 24. Preceding OpId from leader: term: 2 index: 25. (index mismatch)
I20250110 03:07:23.556197 31027 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250110 03:07:23.556326 31027 consensus_queue.cc:1035] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250110 03:07:23.557075 30560 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEADER]: Committing config change with OpId 2.25: config changed from index 24 to 25, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.557878 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5cc5781db40c4148ba1889c45405efc2 with cas_config_opid_index 24: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:23.558018 23229 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.25: config changed from index 24 to 25, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.558692 21468 raft_consensus.cc:2949] T 5cc5781db40c4148ba1889c45405efc2 P ab3ed3d8c2394f78943461e22853c7ac [term 2 FOLLOWER]: Committing config change with OpId 2.25: config changed from index 24 to 25, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } }
I20250110 03:07:23.558725 21188 catalog_manager.cc:5526] T 5cc5781db40c4148ba1889c45405efc2 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 24 to 25, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 2 leader_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:23.559499 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cc5781db40c4148ba1889c45405efc2 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 25) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:23.559615 30983 tablet_replica.cc:331] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:23.559676 30983 raft_consensus.cc:2238] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:23.559762 30983 raft_consensus.cc:2267] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:23.560200 30983 ts_tablet_manager.cc:1905] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:23.561487 30983 ts_tablet_manager.cc:1918] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.24
I20250110 03:07:23.561533 30983 log.cc:1198] T 5cc5781db40c4148ba1889c45405efc2 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:23.561868 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 5cc5781db40c4148ba1889c45405efc2 (table rebalance_test_table_8 [id=3288ec25ec134d5a9c8ee23cfa698e3e]) successfully deleted
I20250110 03:07:23.565299 30863 raft_consensus.cc:1059] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:23.565505 30863 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:23.566027 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250110 03:07:23.566123 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250110 03:07:23.566195 21335 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20250110 03:07:23.566391 31032 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.566534 30862 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.566649 30863 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250110 03:07:23.567610 31032 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.567651 21335 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.567646 23100 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.568678 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.568822 21192 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 19 to 20, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.570596 21468 consensus_queue.cc:237] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:23.571027 22968 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250110 03:07:23.571046 23100 raft_consensus.cc:1270] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20250110 03:07:23.571192 30863 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:23.571326 30863 consensus_queue.cc:1035] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:23.571858 31032 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.572062 22968 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.572144 23100 raft_consensus.cc:2949] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.573290 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b5064ab85103488f873fffef55fd78fd with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:23.573544 21192 catalog_manager.cc:5526] T b5064ab85103488f873fffef55fd78fd P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 20 to 21, VOTER a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.574221 21318 tablet_service.cc:1514] Processing DeleteTablet for tablet b5064ab85103488f873fffef55fd78fd with delete_type TABLET_DATA_TOMBSTONED (TS a2b4fdb2c41d4a7ebf55cfac417c3422 not found in new config with opid_index 21) from {username='slave'} at 127.0.0.1:35500
I20250110 03:07:23.574438 31037 tablet_replica.cc:331] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: stopping tablet replica
I20250110 03:07:23.574530 31037 raft_consensus.cc:2238] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:23.574635 31037 raft_consensus.cc:2267] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:23.574879 31037 ts_tablet_manager.cc:1905] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:23.576628 31037 ts_tablet_manager.cc:1918] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.20
I20250110 03:07:23.576696 31037 log.cc:1198] T b5064ab85103488f873fffef55fd78fd P a2b4fdb2c41d4a7ebf55cfac417c3422: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/b5064ab85103488f873fffef55fd78fd
I20250110 03:07:23.577060 21181 catalog_manager.cc:4872] TS a2b4fdb2c41d4a7ebf55cfac417c3422 (127.14.19.129:45257): tablet b5064ab85103488f873fffef55fd78fd (table rebalance_test_table_2 [id=01ae510e68334126b9e12b8d0c6e771a]) successfully deleted
I20250110 03:07:23.775058 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:23.776854 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:23.777699 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:23.782881 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:23.784466 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:23.787292 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:23.788113 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:23.790871 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:23.827469 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:23.829311 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:23.834339 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:23.834368 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:23.834563 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:23.837284 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:23.839689 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:23.839864 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:23.986578 31142 ts_tablet_manager.cc:927] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Initiating tablet copy from peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:23.986748 31142 tablet_copy_client.cc:323] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.136:42591
I20250110 03:07:23.986949 23370 tablet_copy_service.cc:140] P d0be0af85dee41dd8d01ef4b2af7726f: Received BeginTabletCopySession request for tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer e4d83809ce4746d5ad65e8462857fd9c ({username='slave'} at 127.14.19.133:35405)
I20250110 03:07:23.987020 23370 tablet_copy_service.cc:161] P d0be0af85dee41dd8d01ef4b2af7726f: Beginning new tablet copy session on tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer e4d83809ce4746d5ad65e8462857fd9c at {username='slave'} at 127.14.19.133:35405: session id = e4d83809ce4746d5ad65e8462857fd9c-81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:23.987437 23370 tablet_copy_source_session.cc:215] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:23.987716 31142 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:23.989192 31142 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.18
I20250110 03:07:23.989297 31142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:23.989814 31142 tablet_copy_client.cc:806] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:23.989923 31142 tablet_copy_client.cc:670] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:23.990527 31142 tablet_copy_client.cc:538] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:23.991048 31142 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap starting.
I20250110 03:07:23.992818 23364 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:23.993290 22968 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:23.993441 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250110 03:07:23.993484 28024 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:23.993710 28024 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:23.994385 30890 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.995318 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c5d0922741634918830ee2c163cbce24 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:23.995460 31142 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:23.995414 21188 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 11 to 13, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:23.995677 31142 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Bootstrap complete.
I20250110 03:07:23.995781 31142 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20250110 03:07:23.995857 22968 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.995883 31142 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:23.995868 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:23.995968 31142 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Initialized, Role: LEARNER
I20250110 03:07:23.996001 31142 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 3.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: true } }
I20250110 03:07:23.996102 31142 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:23.996343 23370 tablet_copy_service.cc:342] P d0be0af85dee41dd8d01ef4b2af7726f: Request end of tablet copy session e4d83809ce4746d5ad65e8462857fd9c-81b3bd116ad54a9fa10d2d0c829bb467 received from {username='slave'} at 127.14.19.133:35405
I20250110 03:07:23.996418 23370 tablet_copy_service.cc:434] P d0be0af85dee41dd8d01ef4b2af7726f: ending tablet copy session e4d83809ce4746d5ad65e8462857fd9c-81b3bd116ad54a9fa10d2d0c829bb467 on tablet 81b3bd116ad54a9fa10d2d0c829bb467 with peer e4d83809ce4746d5ad65e8462857fd9c
I20250110 03:07:23.997330 21580 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d0922741634918830ee2c163cbce24 with delete_type TABLET_DATA_TOMBSTONED (TS c57c9dfa476f4d21b79f81114c5d74cb not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49110
I20250110 03:07:23.997529 31146 tablet_replica.cc:331] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: stopping tablet replica
I20250110 03:07:23.997648 31146 raft_consensus.cc:2238] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:23.997741 31146 raft_consensus.cc:2267] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:23.997998 31146 ts_tablet_manager.cc:1905] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:23.999336 31146 ts_tablet_manager.cc:1918] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250110 03:07:23.999387 31146 log.cc:1198] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/c5d0922741634918830ee2c163cbce24
I20250110 03:07:23.999691 21181 catalog_manager.cc:4872] TS c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): tablet c5d0922741634918830ee2c163cbce24 (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:24.043983 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:24.044512 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:24.049306 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:24.049877 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:24.054121 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:24.054464 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:24.055284 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:24.055459 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:24.072701 23364 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:24.073221 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:24.073232 22968 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20250110 03:07:24.073323 23298 consensus_peers.cc:487] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f -> Peer c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131:38011): Couldn't send request to peer c57c9dfa476f4d21b79f81114c5d74cb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:24.073460 28024 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:24.073616 30768 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:24.074306 30768 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:24.074460 22968 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:24.075703 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:24.075791 21196 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 13 to 14, NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) added. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } } }
I20250110 03:07:24.446267 22968 raft_consensus.cc:1212] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.19->[3.20-3.20] Dedup: 3.20->[]
I20250110 03:07:24.529655 31226 ts_tablet_manager.cc:927] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Initiating tablet copy from peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:24.529816 31226 tablet_copy_client.cc:323] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.136:42591
I20250110 03:07:24.530047 23370 tablet_copy_service.cc:140] P d0be0af85dee41dd8d01ef4b2af7726f: Received BeginTabletCopySession request for tablet c5d0922741634918830ee2c163cbce24 from peer c57c9dfa476f4d21b79f81114c5d74cb ({username='slave'} at 127.14.19.131:44395)
I20250110 03:07:24.530112 23370 tablet_copy_service.cc:161] P d0be0af85dee41dd8d01ef4b2af7726f: Beginning new tablet copy session on tablet c5d0922741634918830ee2c163cbce24 from peer c57c9dfa476f4d21b79f81114c5d74cb at {username='slave'} at 127.14.19.131:44395: session id = c57c9dfa476f4d21b79f81114c5d74cb-c5d0922741634918830ee2c163cbce24
I20250110 03:07:24.530664 23370 tablet_copy_source_session.cc:215] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:24.530942 31226 ts_tablet_manager.cc:1905] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:24.532371 31226 ts_tablet_manager.cc:1918] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.12
I20250110 03:07:24.532466 31226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d0922741634918830ee2c163cbce24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:24.532994 31226 tablet_copy_client.cc:806] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:24.533084 31226 tablet_copy_client.cc:670] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:24.533649 31226 tablet_copy_client.cc:538] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:24.534140 31226 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap starting.
I20250110 03:07:24.537142 31226 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:24.537325 31226 tablet_bootstrap.cc:492] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Bootstrap complete.
I20250110 03:07:24.537413 31226 ts_tablet_manager.cc:1397] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:24.537518 31226 raft_consensus.cc:357] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:24.537578 31226 raft_consensus.cc:738] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c57c9dfa476f4d21b79f81114c5d74cb, State: Initialized, Role: LEARNER
I20250110 03:07:24.537612 31226 consensus_queue.cc:260] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: true } }
I20250110 03:07:24.537715 31226 ts_tablet_manager.cc:1428] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:24.537922 23370 tablet_copy_service.cc:342] P d0be0af85dee41dd8d01ef4b2af7726f: Request end of tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-c5d0922741634918830ee2c163cbce24 received from {username='slave'} at 127.14.19.131:44395
I20250110 03:07:24.537977 23370 tablet_copy_service.cc:434] P d0be0af85dee41dd8d01ef4b2af7726f: ending tablet copy session c57c9dfa476f4d21b79f81114c5d74cb-c5d0922741634918830ee2c163cbce24 on tablet c5d0922741634918830ee2c163cbce24 with peer c57c9dfa476f4d21b79f81114c5d74cb
I20250110 03:07:24.927822 21598 raft_consensus.cc:1212] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20250110 03:07:25.056192 31200 raft_consensus.cc:1059] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER e4d83809ce4746d5ad65e8462857fd9c to VOTER
I20250110 03:07:25.056406 31200 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 3.20, Last appended by leader: 17, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:25.056854 22968 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 3 index: 21. (index mismatch)
I20250110 03:07:25.056886 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 3 index: 21. (index mismatch)
I20250110 03:07:25.056901 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 20. Preceding OpId from leader: term: 3 index: 21. (index mismatch)
I20250110 03:07:25.057093 31255 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:25.057199 31200 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:25.057310 28024 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250110 03:07:25.057973 31200 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Committing config change with OpId 3.21: config changed from index 20 to 21, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.058211 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Committing config change with OpId 3.21: config changed from index 20 to 21, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.059056 22968 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Committing config change with OpId 3.21: config changed from index 20 to 21, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.059069 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Committing config change with OpId 3.21: config changed from index 20 to 21, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.060114 21190 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 20 to 21, e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { replace: true } } peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.061326 23364 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 3.21, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:25.061666 22968 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 21. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
I20250110 03:07:25.061682 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 21. Preceding OpId from leader: term: 3 index: 22. (index mismatch)
I20250110 03:07:25.061842 31255 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:25.061914 28024 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250110 03:07:25.062464 28024 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Committing config change with OpId 3.22: config changed from index 21 to 22, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.062639 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Committing config change with OpId 3.22: config changed from index 21 to 22, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.063470 22968 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Committing config change with OpId 3.22: config changed from index 21 to 22, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.063627 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 21: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:25.063723 21189 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d reported cstate change: config changed from index 21 to 22, VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) evicted. New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.064781 21710 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS 9da9d601b7d841b5a8f2c2c53f62a951 not found in new config with opid_index 22) from {username='slave'} at 127.0.0.1:35032
I20250110 03:07:25.064965 31263 tablet_replica.cc:331] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: stopping tablet replica
I20250110 03:07:25.065035 31263 raft_consensus.cc:2238] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:25.065217 31263 raft_consensus.cc:2267] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:25.065450 31263 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:25.066660 31263 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.21
I20250110 03:07:25.066706 31263 log.cc:1198] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/wals/81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:25.067044 21183 catalog_manager.cc:4872] TS 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): tablet 81b3bd116ad54a9fa10d2d0c829bb467 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:25.289505 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:25.289989 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.293020 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:25.298643 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:25.298808 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:25.299145 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.300944 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:25.308961 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:25.321321 21468 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }
I20250110 03:07:25.321882 23231 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:25.321918 21335 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:25.322100 22968 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:25.322113 31331 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:25.322211 31332 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:25.322312 31033 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:25.323038 31033 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.323211 21335 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.324324 22968 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.324586 23231 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.325596 21191 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 10 to 11. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } }
I20250110 03:07:25.327239 21468 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:25.327827 23231 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:25.327955 22968 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:25.328074 21335 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:25.328138 31341 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:25.328400 31033 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250110 03:07:25.328397 21402 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:25.328557 31331 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250110 03:07:25.328634 21182 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet 3b266cc7c4f341f785152932febcb79b with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER failed with leader ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } }.
Pending config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:25.329142 31033 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.329367 21335 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.330338 21182 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3b266cc7c4f341f785152932febcb79b with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:07:25.330605 23231 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.330691 21190 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 reported cstate change: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.331590 21468 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:25.331760 22968 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.332095 21335 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20250110 03:07:25.332253 31341 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:25.332679 22966 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20250110 03:07:25.332871 21402 consensus_peers.cc:487] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac -> Peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): Couldn't send request to peer d0be0af85dee41dd8d01ef4b2af7726f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b266cc7c4f341f785152932febcb79b. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:25.332834 31331 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.333251 31332 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250110 03:07:25.333550 22966 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.333770 21182 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3b266cc7c4f341f785152932febcb79b with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:25.333832 21335 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.334538 21194 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 12 to 13, VOTER b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135) evicted. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } } }
I20250110 03:07:25.337402 23210 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b266cc7c4f341f785152932febcb79b with delete_type TABLET_DATA_TOMBSTONED (TS b2e7bce763e24b63bc760b0b82b53454 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:50748
I20250110 03:07:25.337653 31343 tablet_replica.cc:331] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: stopping tablet replica
I20250110 03:07:25.337760 31343 raft_consensus.cc:2238] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:25.337852 31343 raft_consensus.cc:2267] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:25.338083 31343 ts_tablet_manager.cc:1905] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:25.339430 31343 ts_tablet_manager.cc:1918] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20250110 03:07:25.339478 31343 log.cc:1198] T 3b266cc7c4f341f785152932febcb79b P b2e7bce763e24b63bc760b0b82b53454: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/wal/wals/3b266cc7c4f341f785152932febcb79b
I20250110 03:07:25.339766 21182 catalog_manager.cc:4872] TS b2e7bce763e24b63bc760b0b82b53454 (127.14.19.135:39887): tablet 3b266cc7c4f341f785152932febcb79b (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:25.365947 21182 catalog_manager.cc:5073] ChangeConfig:REMOVE_PEER RPC for tablet 3b266cc7c4f341f785152932febcb79b with cas_config_opid_index 11: aborting the task: latest config opid_index 13; task opid_index 11
I20250110 03:07:25.432050 31255 raft_consensus.cc:1059] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER c57c9dfa476f4d21b79f81114c5d74cb to VOTER
I20250110 03:07:25.432267 31255 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:25.432775 21598 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:25.432857 22966 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:25.432871 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:25.433004 31200 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:25.433111 28024 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:25.433182 31255 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:25.433868 31200 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.434018 21598 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.434203 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.435154 22966 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.435254 21190 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb reported cstate change: config changed from index 14 to 15, c57c9dfa476f4d21b79f81114c5d74cb (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false replace: true } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.436610 23364 consensus_queue.cc:237] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } }
I20250110 03:07:25.436986 21598 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:25.437022 21335 raft_consensus.cc:1270] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:25.437178 31200 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:25.437274 31255 consensus_queue.cc:1035] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:25.437950 31255 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.438125 21335 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.438189 21598 raft_consensus.cc:2949] T c5d0922741634918830ee2c163cbce24 P c57c9dfa476f4d21b79f81114c5d74cb [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } } }
I20250110 03:07:25.439060 21180 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c5d0922741634918830ee2c163cbce24 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:25.439195 21194 catalog_manager.cc:5526] T c5d0922741634918830ee2c163cbce24 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 15 to 16, VOTER e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133) evicted. New cstate: current_term: 2 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c57c9dfa476f4d21b79f81114c5d74cb" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 38011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:25.439970 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet c5d0922741634918830ee2c163cbce24 with delete_type TABLET_DATA_TOMBSTONED (TS e4d83809ce4746d5ad65e8462857fd9c not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:49544
I20250110 03:07:25.440142 31348 tablet_replica.cc:331] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: stopping tablet replica
I20250110 03:07:25.440230 31348 raft_consensus.cc:2238] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:25.440317 31348 raft_consensus.cc:2267] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:25.440533 31348 ts_tablet_manager.cc:1905] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:25.441744 31348 ts_tablet_manager.cc:1918] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250110 03:07:25.441802 31348 log.cc:1198] T c5d0922741634918830ee2c163cbce24 P e4d83809ce4746d5ad65e8462857fd9c: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/c5d0922741634918830ee2c163cbce24
I20250110 03:07:25.442157 21181 catalog_manager.cc:4872] TS e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091): tablet c5d0922741634918830ee2c163cbce24 (table rebalance_test_table_0 [id=9996d4b6c27c4133a444d6f8fc722e5a]) successfully deleted
I20250110 03:07:25.560078 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:25.560410 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.569087 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:25.569276 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:25.569782 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:25.570981 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:25.575030 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.578783 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:25.590751 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:25.594502 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:25.595371 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:25.595731 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:25.596431 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.599893 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:25.606688 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:25.610028 23364 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 3.22, Last appended by leader: 17, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:25.610585 22966 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 22. Preceding OpId from leader: term: 3 index: 23. (index mismatch)
I20250110 03:07:25.610589 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 22. Preceding OpId from leader: term: 3 index: 23. (index mismatch)
I20250110 03:07:25.610814 30890 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
W20250110 03:07:25.610910 23300 consensus_peers.cc:487] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:25.610939 30890 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250110 03:07:25.611562 30890 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Committing config change with OpId 3.23: config changed from index 22 to 23, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.612521 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.613001 21188 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 22 to 23, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:25.613065 22966 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Committing config change with OpId 3.23: config changed from index 22 to 23, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.613996 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Committing config change with OpId 3.23: config changed from index 22 to 23, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.652143 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:25.653815 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.660931 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:25.661062 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:25.661814 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:25.662019 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:25.662629 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:25.669236 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:25.683182 22966 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:25.683714 23364 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250110 03:07:25.683776 23231 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20250110 03:07:25.683812 22907 consensus_peers.cc:487] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c -> Peer 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355): Couldn't send request to peer 9da9d601b7d841b5a8f2c2c53f62a951. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:25.683982 31496 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:25.684119 31145 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:25.684804 31496 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.684895 23364 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.684976 23231 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.686182 21191 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 13 to 14, NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) added. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } } }
I20250110 03:07:25.947026 31515 ts_tablet_manager.cc:927] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: Initiating tablet copy from peer ab3ed3d8c2394f78943461e22853c7ac (127.14.19.130:35469)
I20250110 03:07:25.947222 31515 tablet_copy_client.cc:323] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:35469
I20250110 03:07:25.947451 21488 tablet_copy_service.cc:140] P ab3ed3d8c2394f78943461e22853c7ac: Received BeginTabletCopySession request for tablet 3b266cc7c4f341f785152932febcb79b from peer d0be0af85dee41dd8d01ef4b2af7726f ({username='slave'} at 127.14.19.136:48651)
I20250110 03:07:25.947501 21488 tablet_copy_service.cc:161] P ab3ed3d8c2394f78943461e22853c7ac: Beginning new tablet copy session on tablet 3b266cc7c4f341f785152932febcb79b from peer d0be0af85dee41dd8d01ef4b2af7726f at {username='slave'} at 127.14.19.136:48651: session id = d0be0af85dee41dd8d01ef4b2af7726f-3b266cc7c4f341f785152932febcb79b
I20250110 03:07:25.947845 21488 tablet_copy_source_session.cc:215] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:25.948174 31515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b266cc7c4f341f785152932febcb79b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:25.949190 31515 tablet_copy_client.cc:806] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:25.949323 31515 tablet_copy_client.cc:670] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:25.949914 31515 tablet_copy_client.cc:538] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:25.950500 31515 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap starting.
I20250110 03:07:25.954217 31515 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:25.954372 31515 tablet_bootstrap.cc:492] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: Bootstrap complete.
I20250110 03:07:25.954440 31515 ts_tablet_manager.cc:1397] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250110 03:07:25.954524 31515 raft_consensus.cc:357] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:25.954623 31515 raft_consensus.cc:738] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Initialized, Role: LEARNER
I20250110 03:07:25.954654 31515 consensus_queue.cc:260] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: NON_VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: true } }
I20250110 03:07:25.954754 31515 ts_tablet_manager.cc:1428] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20250110 03:07:25.955034 21488 tablet_copy_service.cc:342] P ab3ed3d8c2394f78943461e22853c7ac: Request end of tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-3b266cc7c4f341f785152932febcb79b received from {username='slave'} at 127.14.19.136:48651
I20250110 03:07:25.955101 21488 tablet_copy_service.cc:434] P ab3ed3d8c2394f78943461e22853c7ac: ending tablet copy session d0be0af85dee41dd8d01ef4b2af7726f-3b266cc7c4f341f785152932febcb79b on tablet 3b266cc7c4f341f785152932febcb79b with peer d0be0af85dee41dd8d01ef4b2af7726f
I20250110 03:07:26.006374 31518 ts_tablet_manager.cc:927] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591)
I20250110 03:07:26.006527 31518 tablet_copy_client.cc:323] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.136:42591
I20250110 03:07:26.006747 23370 tablet_copy_service.cc:140] P d0be0af85dee41dd8d01ef4b2af7726f: Received BeginTabletCopySession request for tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:34013)
I20250110 03:07:26.006801 23370 tablet_copy_service.cc:161] P d0be0af85dee41dd8d01ef4b2af7726f: Beginning new tablet copy session on tablet 81b3bd116ad54a9fa10d2d0c829bb467 from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:34013: session id = 9da9d601b7d841b5a8f2c2c53f62a951-81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:26.007174 23370 tablet_copy_source_session.cc:215] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:26.007457 31518 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:26.009032 31518 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.21
I20250110 03:07:26.009143 31518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b3bd116ad54a9fa10d2d0c829bb467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:26.009886 31518 tablet_copy_client.cc:806] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:26.009984 31518 tablet_copy_client.cc:670] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:26.010608 31518 tablet_copy_client.cc:538] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:26.011147 31518 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:26.014360 31518 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=0 applied=23 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:26.014539 31518 tablet_bootstrap.cc:492] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:26.014657 31518 ts_tablet_manager.cc:1397] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20250110 03:07:26.014757 31518 raft_consensus.cc:357] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:26.014817 31518 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:26.014842 31518 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 3.23, Last appended by leader: 23, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:26.014933 31518 ts_tablet_manager.cc:1428] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:26.015110 23370 tablet_copy_service.cc:342] P d0be0af85dee41dd8d01ef4b2af7726f: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-81b3bd116ad54a9fa10d2d0c829bb467 received from {username='slave'} at 127.14.19.132:34013
I20250110 03:07:26.015170 23370 tablet_copy_service.cc:434] P d0be0af85dee41dd8d01ef4b2af7726f: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-81b3bd116ad54a9fa10d2d0c829bb467 on tablet 81b3bd116ad54a9fa10d2d0c829bb467 with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:26.141592 31518 ts_tablet_manager.cc:927] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Initiating tablet copy from peer e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133:42091)
I20250110 03:07:26.141773 31518 tablet_copy_client.cc:323] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.133:42091
I20250110 03:07:26.141995 22991 tablet_copy_service.cc:140] P e4d83809ce4746d5ad65e8462857fd9c: Received BeginTabletCopySession request for tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer 9da9d601b7d841b5a8f2c2c53f62a951 ({username='slave'} at 127.14.19.132:39361)
I20250110 03:07:26.142060 22991 tablet_copy_service.cc:161] P e4d83809ce4746d5ad65e8462857fd9c: Beginning new tablet copy session on tablet 0760e9b8e3c642f09c3287ca586a2a1d from peer 9da9d601b7d841b5a8f2c2c53f62a951 at {username='slave'} at 127.14.19.132:39361: session id = 9da9d601b7d841b5a8f2c2c53f62a951-0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:26.142593 22991 tablet_copy_source_session.cc:215] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:26.142917 31518 ts_tablet_manager.cc:1905] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:26.144488 31518 ts_tablet_manager.cc:1918] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20250110 03:07:26.144579 31518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0760e9b8e3c642f09c3287ca586a2a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:26.145062 31518 tablet_copy_client.cc:806] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:26.145167 31518 tablet_copy_client.cc:670] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:26.145716 31518 tablet_copy_client.cc:538] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:26.146163 31518 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap starting.
I20250110 03:07:26.148825 31518 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:26.148973 31518 tablet_bootstrap.cc:492] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Bootstrap complete.
I20250110 03:07:26.149039 31518 ts_tablet_manager.cc:1397] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:26.149120 31518 raft_consensus.cc:357] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:26.149178 31518 raft_consensus.cc:738] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9da9d601b7d841b5a8f2c2c53f62a951, State: Initialized, Role: LEARNER
I20250110 03:07:26.149199 31518 consensus_queue.cc:260] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: true } }
I20250110 03:07:26.149277 31518 ts_tablet_manager.cc:1428] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:26.149492 22991 tablet_copy_service.cc:342] P e4d83809ce4746d5ad65e8462857fd9c: Request end of tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-0760e9b8e3c642f09c3287ca586a2a1d received from {username='slave'} at 127.14.19.132:39361
I20250110 03:07:26.149549 22991 tablet_copy_service.cc:434] P e4d83809ce4746d5ad65e8462857fd9c: ending tablet copy session 9da9d601b7d841b5a8f2c2c53f62a951-0760e9b8e3c642f09c3287ca586a2a1d on tablet 0760e9b8e3c642f09c3287ca586a2a1d with peer 9da9d601b7d841b5a8f2c2c53f62a951
I20250110 03:07:26.429502 23364 raft_consensus.cc:1212] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20250110 03:07:26.484876 21730 raft_consensus.cc:1212] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.22->[3.23-3.23] Dedup: 3.23->[]
I20250110 03:07:26.668140 21730 raft_consensus.cc:1212] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20250110 03:07:27.002141 31549 raft_consensus.cc:1059] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:27.002316 31549 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 3.23, Last appended by leader: 17, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:27.002774 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 LEARNER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 23. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20250110 03:07:27.002839 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 23. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20250110 03:07:27.002923 22966 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Refusing update from remote peer d0be0af85dee41dd8d01ef4b2af7726f: Log matching property violated. Preceding OpId in replica: term: 3 index: 23. Preceding OpId from leader: term: 3 index: 24. (index mismatch)
I20250110 03:07:27.003037 31580 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:27.003171 31577 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:27.003381 31579 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250110 03:07:27.004103 31580 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Committing config change with OpId 3.24: config changed from index 23 to 24, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.004220 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Committing config change with OpId 3.24: config changed from index 23 to 24, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.005175 22966 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Committing config change with OpId 3.24: config changed from index 23 to 24, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.005273 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Committing config change with OpId 3.24: config changed from index 23 to 24, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.006312 21194 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f reported cstate change: config changed from index 23 to 24, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:27.044005 31543 raft_consensus.cc:1059] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac: attempting to promote NON_VOTER d0be0af85dee41dd8d01ef4b2af7726f to VOTER
I20250110 03:07:27.044201 31543 consensus_queue.cc:237] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } }
I20250110 03:07:27.044659 23364 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f [term 1 LEARNER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:27.044724 22966 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:27.044736 21335 raft_consensus.cc:1270] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Refusing update from remote peer ab3ed3d8c2394f78943461e22853c7ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20250110 03:07:27.044907 31594 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:27.045011 31597 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:27.045140 31543 consensus_queue.cc:1035] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250110 03:07:27.046298 31543 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P ab3ed3d8c2394f78943461e22853c7ac [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:27.046434 22966 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:27.046614 23362 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P d0be0af85dee41dd8d01ef4b2af7726f [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:27.047645 21335 raft_consensus.cc:2949] T 3b266cc7c4f341f785152932febcb79b P a2b4fdb2c41d4a7ebf55cfac417c3422 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:27.047719 21191 catalog_manager.cc:5526] T 3b266cc7c4f341f785152932febcb79b P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 13 to 14, d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab3ed3d8c2394f78943461e22853c7ac" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a2b4fdb2c41d4a7ebf55cfac417c3422" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 45257 } } peers { permanent_uuid: "ab3ed3d8c2394f78943461e22853c7ac" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 35469 } attrs { replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false } } }
I20250110 03:07:27.051025 23362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51594
I20250110 03:07:27.051092 23362 raft_consensus.cc:604] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Received request to transfer leadership
I20250110 03:07:27.213318 31561 raft_consensus.cc:1059] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c: attempting to promote NON_VOTER 9da9d601b7d841b5a8f2c2c53f62a951 to VOTER
I20250110 03:07:27.213529 31561 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:27.214000 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 LEARNER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:27.214042 23362 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:27.214136 23231 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20250110 03:07:27.214226 31613 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:27.214394 31611 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:27.214583 31614 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250110 03:07:27.215444 31613 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.215574 23362 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.216666 23231 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.216864 21188 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:27.216912 21730 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.217794 22966 consensus_queue.cc:237] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:27.218489 21730 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:27.218626 23231 raft_consensus.cc:1270] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250110 03:07:27.218725 31561 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:27.218832 31613 consensus_queue.cc:1035] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250110 03:07:27.219295 31614 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.219539 23231 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P b2e7bce763e24b63bc760b0b82b53454 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.220244 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0760e9b8e3c642f09c3287ca586a2a1d with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:27.220363 21730 raft_consensus.cc:2949] T 0760e9b8e3c642f09c3287ca586a2a1d P 9da9d601b7d841b5a8f2c2c53f62a951 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:27.220450 21190 catalog_manager.cc:5526] T 0760e9b8e3c642f09c3287ca586a2a1d P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 15 to 16, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 2 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7bce763e24b63bc760b0b82b53454" member_type: VOTER last_known_addr { host: "127.14.19.135" port: 39887 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:27.221217 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 0760e9b8e3c642f09c3287ca586a2a1d with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:27.221483 31619 tablet_replica.cc:331] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:27.221555 31619 raft_consensus.cc:2238] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:27.221665 31619 raft_consensus.cc:2267] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:27.221886 31619 ts_tablet_manager.cc:1905] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:27.222975 31619 ts_tablet_manager.cc:1918] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20250110 03:07:27.223024 31619 log.cc:1198] T 0760e9b8e3c642f09c3287ca586a2a1d P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:27.223340 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 0760e9b8e3c642f09c3287ca586a2a1d (table rebalance_test_table_9 [id=db5dbf2354c14d9891dd1c7ae6ed2f24]) successfully deleted
I20250110 03:07:27.258066 23362 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51608
I20250110 03:07:27.258155 23362 raft_consensus.cc:604] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Received request to transfer leadership
I20250110 03:07:27.350804 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:27.351145 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:27.354785 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:27.357887 23344 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:27.359078 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:27.359864 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:27.363602 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:27.369972 22951 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:27.404472 31580 raft_consensus.cc:988] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: : Instructing follower e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:27.404553 31580 raft_consensus.cc:1076] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Signalling peer e4d83809ce4746d5ad65e8462857fd9c to start an election
I20250110 03:07:27.404883 22966 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467"
dest_uuid: "e4d83809ce4746d5ad65e8462857fd9c"
from {username='slave'} at 127.14.19.136:42983
I20250110 03:07:27.404954 22966 raft_consensus.cc:491] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:27.404987 22966 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 3 FOLLOWER]: Advancing to term 4
I20250110 03:07:27.405779 22966 raft_consensus.cc:513] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:27.405937 22966 leader_election.cc:290] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 4 election: Requested vote from peers 531cf07bcce6435c9e18185c60b2118d (127.14.19.134:45351), d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591), 9da9d601b7d841b5a8f2c2c53f62a951 (127.14.19.132:45355)
I20250110 03:07:27.406203 23100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: true dest_uuid: "531cf07bcce6435c9e18185c60b2118d"
I20250110 03:07:27.406297 23100 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 3 FOLLOWER]: Advancing to term 4
I20250110 03:07:27.407107 23100 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 4.
I20250110 03:07:27.407337 23362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: true dest_uuid: "d0be0af85dee41dd8d01ef4b2af7726f"
I20250110 03:07:27.407415 23362 raft_consensus.cc:3049] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Stepping down as leader of term 3
I20250110 03:07:27.407444 23362 raft_consensus.cc:738] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d0be0af85dee41dd8d01ef4b2af7726f, State: Running, Role: LEADER
I20250110 03:07:27.407529 23362 consensus_queue.cc:260] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 3.24, Last appended by leader: 24, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:27.407641 23362 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 3 FOLLOWER]: Advancing to term 4
I20250110 03:07:27.408059 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:27.408365 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:27.408429 23362 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 4.
I20250110 03:07:27.409166 22905 leader_election.cc:304] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 531cf07bcce6435c9e18185c60b2118d, d0be0af85dee41dd8d01ef4b2af7726f, e4d83809ce4746d5ad65e8462857fd9c; no voters:
I20250110 03:07:27.409272 31613 raft_consensus.cc:2798] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 4 FOLLOWER]: Leader election won for term 4
I20250110 03:07:27.409314 31613 raft_consensus.cc:695] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 4 LEADER]: Becoming Leader. State: Replica: e4d83809ce4746d5ad65e8462857fd9c, State: Running, Role: LEADER
I20250110 03:07:27.409354 31613 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 3.24, Last appended by leader: 24, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:27.409996 21194 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: term changed from 3 to 4, leader changed from d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) to e4d83809ce4746d5ad65e8462857fd9c (127.14.19.133). New cstate: current_term: 4 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:27.410876 21730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b3bd116ad54a9fa10d2d0c829bb467" candidate_uuid: "e4d83809ce4746d5ad65e8462857fd9c" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: true dest_uuid: "9da9d601b7d841b5a8f2c2c53f62a951"
I20250110 03:07:27.410952 21730 raft_consensus.cc:3054] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 3 FOLLOWER]: Advancing to term 4
I20250110 03:07:27.411612 21730 raft_consensus.cc:2463] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4d83809ce4746d5ad65e8462857fd9c in term 4.
I20250110 03:07:27.414506 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:27.415300 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:27.416322 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:27.416913 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:27.422493 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:27.424063 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:27.920428 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 4 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 24. Preceding OpId from leader: term: 4 index: 25. (index mismatch)
I20250110 03:07:27.920686 31611 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250110 03:07:27.986179 23362 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 4 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 24. Preceding OpId from leader: term: 4 index: 25. (index mismatch)
I20250110 03:07:27.986450 31611 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0be0af85dee41dd8d01ef4b2af7726f" member_type: VOTER last_known_addr { host: "127.14.19.136" port: 42591 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250110 03:07:27.987613 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 4 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 3 index: 24. Preceding OpId from leader: term: 4 index: 25. (index mismatch)
I20250110 03:07:27.987875 31735 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250110 03:07:28.506922 22966 consensus_queue.cc:237] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 4.25, Last appended by leader: 24, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } }
I20250110 03:07:28.507423 21730 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 4 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 25. Preceding OpId from leader: term: 4 index: 26. (index mismatch)
I20250110 03:07:28.507468 23100 raft_consensus.cc:1270] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 4 FOLLOWER]: Refusing update from remote peer e4d83809ce4746d5ad65e8462857fd9c: Log matching property violated. Preceding OpId in replica: term: 4 index: 25. Preceding OpId from leader: term: 4 index: 26. (index mismatch)
I20250110 03:07:28.507710 31754 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250110 03:07:28.507836 31735 consensus_queue.cc:1035] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250110 03:07:28.508590 31735 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c [term 4 LEADER]: Committing config change with OpId 4.26: config changed from index 24 to 26, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:28.508751 21730 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 9da9d601b7d841b5a8f2c2c53f62a951 [term 4 FOLLOWER]: Committing config change with OpId 4.26: config changed from index 24 to 26, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:28.508770 23100 raft_consensus.cc:2949] T 81b3bd116ad54a9fa10d2d0c829bb467 P 531cf07bcce6435c9e18185c60b2118d [term 4 FOLLOWER]: Committing config change with OpId 4.26: config changed from index 24 to 26, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } } }
I20250110 03:07:28.509958 21194 catalog_manager.cc:5526] T 81b3bd116ad54a9fa10d2d0c829bb467 P e4d83809ce4746d5ad65e8462857fd9c reported cstate change: config changed from index 24 to 26, VOTER d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136) evicted. New cstate: current_term: 4 leader_uuid: "e4d83809ce4746d5ad65e8462857fd9c" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "531cf07bcce6435c9e18185c60b2118d" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 45351 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d83809ce4746d5ad65e8462857fd9c" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 42091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9da9d601b7d841b5a8f2c2c53f62a951" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 45355 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:28.510104 21181 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with cas_config_opid_index 24: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:28.511147 23344 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b3bd116ad54a9fa10d2d0c829bb467 with delete_type TABLET_DATA_TOMBSTONED (TS d0be0af85dee41dd8d01ef4b2af7726f not found in new config with opid_index 26) from {username='slave'} at 127.0.0.1:59590
I20250110 03:07:28.511368 31761 tablet_replica.cc:331] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: stopping tablet replica
I20250110 03:07:28.511476 31761 raft_consensus.cc:2238] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 4 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:28.511529 31761 raft_consensus.cc:2267] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:28.511811 31761 ts_tablet_manager.cc:1905] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:28.513154 31761 ts_tablet_manager.cc:1918] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.25
I20250110 03:07:28.513221 31761 log.cc:1198] T 81b3bd116ad54a9fa10d2d0c829bb467 P d0be0af85dee41dd8d01ef4b2af7726f: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/wal/wals/81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:28.513495 21180 catalog_manager.cc:4872] TS d0be0af85dee41dd8d01ef4b2af7726f (127.14.19.136:42591): tablet 81b3bd116ad54a9fa10d2d0c829bb467 (table rebalance_test_table_4 [id=df60da805edf4c46b41882491c27630b]) successfully deleted
I20250110 03:07:28.713358 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:28.714082 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.715782 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:28.722690 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:28.723492 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:28.728300 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:28.728489 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.728763 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.768292 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:28.769851 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.769920 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:28.772615 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.775013 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:28.777328 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.783936 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:28.784324 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:28.828946 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.830049 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:28.830899 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.831983 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:28.838656 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:28.840196 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:28.840510 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:28.846033 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f0d1744b334c4bae9f757319e7a75e38 | 127.14.19.190:41945 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:35525 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
531cf07bcce6435c9e18185c60b2118d | 127.14.19.134:45351 | HEALTHY | <none> | 0 | 0
9da9d601b7d841b5a8f2c2c53f62a951 | 127.14.19.132:45355 | HEALTHY | <none> | 3 | 0
a2b4fdb2c41d4a7ebf55cfac417c3422 | 127.14.19.129:45257 | HEALTHY | <none> | 3 | 0
ab3ed3d8c2394f78943461e22853c7ac | 127.14.19.130:35469 | HEALTHY | <none> | 10 | 0
b2e7bce763e24b63bc760b0b82b53454 | 127.14.19.135:39887 | HEALTHY | <none> | 1 | 0
c57c9dfa476f4d21b79f81114c5d74cb | 127.14.19.131:38011 | HEALTHY | <none> | 6 | 0
d0be0af85dee41dd8d01ef4b2af7726f | 127.14.19.136:42591 | HEALTHY | <none> | 4 | 0
e4d83809ce4746d5ad65e8462857fd9c | 127.14.19.133:42091 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.14.19.129 | experimental | 127.14.19.129:45257
local_ip_for_outbound_sockets | 127.14.19.130 | experimental | 127.14.19.130:35469
local_ip_for_outbound_sockets | 127.14.19.131 | experimental | 127.14.19.131:38011
local_ip_for_outbound_sockets | 127.14.19.132 | experimental | 127.14.19.132:45355
local_ip_for_outbound_sockets | 127.14.19.133 | experimental | 127.14.19.133:42091
local_ip_for_outbound_sockets | 127.14.19.134 | experimental | 127.14.19.134:45351
local_ip_for_outbound_sockets | 127.14.19.135 | experimental | 127.14.19.135:39887
local_ip_for_outbound_sockets | 127.14.19.136 | experimental | 127.14.19.136:42591
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.19.129:45257
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.19.130:35469
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.19.131:38011
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.19.132:45355
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.19.133:42091
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.19.134:45351
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.19.135:39887
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.14.19.136:42591
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:35525 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 12
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 10
Tablets | 30
Replicas | 92
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 03:07:28.857048 14414 log_verifier.cc:126] Checking tablet 0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:28.875900 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet 0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:28.875980 14414 log_verifier.cc:126] Checking tablet 122762e845a046f4a84afeeacc01b42f
I20250110 03:07:28.893740 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 122762e845a046f4a84afeeacc01b42f
I20250110 03:07:28.893800 14414 log_verifier.cc:126] Checking tablet 22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:28.911298 14414 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:28.911415 14414 log_verifier.cc:126] Checking tablet 250f547c54a84a35b2850d50b897810e
I20250110 03:07:28.930095 14414 log_verifier.cc:177] Verified matching terms for 21 ops in tablet 250f547c54a84a35b2850d50b897810e
I20250110 03:07:28.930153 14414 log_verifier.cc:126] Checking tablet 32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:28.948472 14414 log_verifier.cc:177] Verified matching terms for 24 ops in tablet 32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:28.948544 14414 log_verifier.cc:126] Checking tablet 3b266cc7c4f341f785152932febcb79b
I20250110 03:07:28.973150 14414 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 3b266cc7c4f341f785152932febcb79b
I20250110 03:07:28.973210 14414 log_verifier.cc:126] Checking tablet 41db68f089644324af518422d2cad2fc
I20250110 03:07:28.991333 14414 log_verifier.cc:177] Verified matching terms for 23 ops in tablet 41db68f089644324af518422d2cad2fc
I20250110 03:07:28.991389 14414 log_verifier.cc:126] Checking tablet 41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:29.009291 14414 log_verifier.cc:177] Verified matching terms for 18 ops in tablet 41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:29.009347 14414 log_verifier.cc:126] Checking tablet 44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:29.028277 14414 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:29.028340 14414 log_verifier.cc:126] Checking tablet 49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:29.047009 14414 log_verifier.cc:177] Verified matching terms for 15 ops in tablet 49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:29.047062 14414 log_verifier.cc:126] Checking tablet 4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:29.064848 14414 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:29.064905 14414 log_verifier.cc:126] Checking tablet 58e33e63f48f4b758c19774888345ee0
I20250110 03:07:29.082409 14414 log_verifier.cc:177] Verified matching terms for 15 ops in tablet 58e33e63f48f4b758c19774888345ee0
I20250110 03:07:29.082471 14414 log_verifier.cc:126] Checking tablet 5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:29.099624 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:29.099705 14414 log_verifier.cc:126] Checking tablet 5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:29.118861 14414 log_verifier.cc:177] Verified matching terms for 25 ops in tablet 5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:29.118921 14414 log_verifier.cc:126] Checking tablet 62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:29.137013 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet 62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:29.137073 14414 log_verifier.cc:126] Checking tablet 7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:29.155162 14414 log_verifier.cc:177] Verified matching terms for 17 ops in tablet 7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:29.155218 14414 log_verifier.cc:126] Checking tablet 80152844286545c68c7656ef9bbe8604
I20250110 03:07:29.173187 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 80152844286545c68c7656ef9bbe8604
I20250110 03:07:29.173254 14414 log_verifier.cc:126] Checking tablet 81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:29.192142 14414 log_verifier.cc:177] Verified matching terms for 26 ops in tablet 81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:29.192206 14414 log_verifier.cc:126] Checking tablet 87f8afb021a7407089c710f6dd588d03
I20250110 03:07:29.209966 14414 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 87f8afb021a7407089c710f6dd588d03
I20250110 03:07:29.210037 14414 log_verifier.cc:126] Checking tablet 9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:29.227979 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:29.228044 14414 log_verifier.cc:126] Checking tablet a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:29.245481 14414 log_verifier.cc:177] Verified matching terms for 8 ops in tablet a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:29.245545 14414 log_verifier.cc:126] Checking tablet a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:29.262995 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:29.263047 14414 log_verifier.cc:126] Checking tablet abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:29.280616 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:29.280673 14414 log_verifier.cc:126] Checking tablet b5064ab85103488f873fffef55fd78fd
I20250110 03:07:29.298786 14414 log_verifier.cc:177] Verified matching terms for 21 ops in tablet b5064ab85103488f873fffef55fd78fd
I20250110 03:07:29.298918 14414 log_verifier.cc:126] Checking tablet c5d0922741634918830ee2c163cbce24
I20250110 03:07:29.322479 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet c5d0922741634918830ee2c163cbce24
I20250110 03:07:29.322568 14414 log_verifier.cc:126] Checking tablet d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:29.348487 14414 log_verifier.cc:177] Verified matching terms for 15 ops in tablet d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:29.348559 14414 log_verifier.cc:126] Checking tablet ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:29.374773 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:29.374850 14414 log_verifier.cc:126] Checking tablet ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:29.400035 14414 log_verifier.cc:177] Verified matching terms for 9 ops in tablet ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:29.400115 14414 log_verifier.cc:126] Checking tablet f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:29.431483 14414 log_verifier.cc:177] Verified matching terms for 17 ops in tablet f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:29.431555 14414 log_verifier.cc:126] Checking tablet fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:29.449718 14414 log_verifier.cc:177] Verified matching terms for 17 ops in tablet fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:29.564638 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.565107 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:29.565546 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.570595 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:29.570662 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:29.572742 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.575183 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:29.585074 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:29.627243 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:29.627490 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.631373 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.635926 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:29.641489 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:29.641742 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:29.642007 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:29.645861 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.701246 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:29.704525 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.706588 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:29.707798 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:29.711242 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:29.711833 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:29.720471 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.721531 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.848275 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:29.850080 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.854364 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:29.857246 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.857584 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:29.859181 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.866519 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:29.867440 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:29.909817 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.912443 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.914831 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:29.918370 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:29.919906 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:29.924417 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.925000 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:29.926478 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:29.985731 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:29.990098 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:29.991642 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:29.991739 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:29.996850 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:29.999783 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:30.000634 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:30.003783 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:30.048728 21710 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:30.049486 21447 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20250110 03:07:30.055620 21580 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:30.056114 23082 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:30.057361 21318 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:30.057610 23344 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:30.065948 22951 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:30.066906 23210 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
f0d1744b334c4bae9f757319e7a75e38 | 127.14.19.190:41945 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:35525 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
531cf07bcce6435c9e18185c60b2118d | 127.14.19.134:45351 | HEALTHY | <none> | 0 | 0
9da9d601b7d841b5a8f2c2c53f62a951 | 127.14.19.132:45355 | HEALTHY | <none> | 3 | 0
a2b4fdb2c41d4a7ebf55cfac417c3422 | 127.14.19.129:45257 | HEALTHY | <none> | 3 | 0
ab3ed3d8c2394f78943461e22853c7ac | 127.14.19.130:35469 | HEALTHY | <none> | 10 | 0
b2e7bce763e24b63bc760b0b82b53454 | 127.14.19.135:39887 | HEALTHY | <none> | 1 | 0
c57c9dfa476f4d21b79f81114c5d74cb | 127.14.19.131:38011 | HEALTHY | <none> | 6 | 0
d0be0af85dee41dd8d01ef4b2af7726f | 127.14.19.136:42591 | HEALTHY | <none> | 4 | 0
e4d83809ce4746d5ad65e8462857fd9c | 127.14.19.133:42091 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.14.19.129 | experimental | 127.14.19.129:45257
local_ip_for_outbound_sockets | 127.14.19.130 | experimental | 127.14.19.130:35469
local_ip_for_outbound_sockets | 127.14.19.131 | experimental | 127.14.19.131:38011
local_ip_for_outbound_sockets | 127.14.19.132 | experimental | 127.14.19.132:45355
local_ip_for_outbound_sockets | 127.14.19.133 | experimental | 127.14.19.133:42091
local_ip_for_outbound_sockets | 127.14.19.134 | experimental | 127.14.19.134:45351
local_ip_for_outbound_sockets | 127.14.19.135 | experimental | 127.14.19.135:39887
local_ip_for_outbound_sockets | 127.14.19.136 | experimental | 127.14.19.136:42591
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.19.129:45257
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.19.130:35469
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.19.131:38011
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.19.132:45355
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.14.19.133:42091
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.14.19.134:45351
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.14.19.135:39887
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.14.19.136:42591
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:35525 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 12
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 10
Tablets | 30
Replicas | 92
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 03:07:30.076491 14414 log_verifier.cc:126] Checking tablet 0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:30.087797 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet 0760e9b8e3c642f09c3287ca586a2a1d
I20250110 03:07:30.087843 14414 log_verifier.cc:126] Checking tablet 122762e845a046f4a84afeeacc01b42f
I20250110 03:07:30.098174 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 122762e845a046f4a84afeeacc01b42f
I20250110 03:07:30.098218 14414 log_verifier.cc:126] Checking tablet 22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:30.107959 14414 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 22fe8e3aabc64a848ffc1e4f21fa1b51
I20250110 03:07:30.108017 14414 log_verifier.cc:126] Checking tablet 250f547c54a84a35b2850d50b897810e
I20250110 03:07:30.118412 14414 log_verifier.cc:177] Verified matching terms for 21 ops in tablet 250f547c54a84a35b2850d50b897810e
I20250110 03:07:30.118453 14414 log_verifier.cc:126] Checking tablet 32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:30.129278 14414 log_verifier.cc:177] Verified matching terms for 24 ops in tablet 32609d619a0242d1ba0a851fa0a7812d
I20250110 03:07:30.129326 14414 log_verifier.cc:126] Checking tablet 3b266cc7c4f341f785152932febcb79b
I20250110 03:07:30.142966 14414 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 3b266cc7c4f341f785152932febcb79b
I20250110 03:07:30.143014 14414 log_verifier.cc:126] Checking tablet 41db68f089644324af518422d2cad2fc
I20250110 03:07:30.153622 14414 log_verifier.cc:177] Verified matching terms for 23 ops in tablet 41db68f089644324af518422d2cad2fc
I20250110 03:07:30.153676 14414 log_verifier.cc:126] Checking tablet 41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:30.163888 14414 log_verifier.cc:177] Verified matching terms for 18 ops in tablet 41fa30056c0a4b30ab8fcaa14645e7a4
I20250110 03:07:30.163931 14414 log_verifier.cc:126] Checking tablet 44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:30.174135 14414 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 44bb3816ff5a4d44a249b8bbabdc7b04
I20250110 03:07:30.174180 14414 log_verifier.cc:126] Checking tablet 49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:30.184129 14414 log_verifier.cc:177] Verified matching terms for 15 ops in tablet 49c3c5c20dcc4474b207b9c73904219f
I20250110 03:07:30.184170 14414 log_verifier.cc:126] Checking tablet 4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:30.194077 14414 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 4e11143dad6b402fb0d16c48285646c9
I20250110 03:07:30.194125 14414 log_verifier.cc:126] Checking tablet 58e33e63f48f4b758c19774888345ee0
I20250110 03:07:30.204000 14414 log_verifier.cc:177] Verified matching terms for 15 ops in tablet 58e33e63f48f4b758c19774888345ee0
I20250110 03:07:30.204048 14414 log_verifier.cc:126] Checking tablet 5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:30.213950 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 5a1a954889a9452f8a8ee55aa0a6f26e
I20250110 03:07:30.213994 14414 log_verifier.cc:126] Checking tablet 5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:30.224884 14414 log_verifier.cc:177] Verified matching terms for 25 ops in tablet 5cc5781db40c4148ba1889c45405efc2
I20250110 03:07:30.224931 14414 log_verifier.cc:126] Checking tablet 62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:30.235301 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet 62aacc8deebd43debd76a2c10815da2f
I20250110 03:07:30.235345 14414 log_verifier.cc:126] Checking tablet 7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:30.245986 14414 log_verifier.cc:177] Verified matching terms for 17 ops in tablet 7dbef1612c8b4e2baa670b1cbe0e8960
I20250110 03:07:30.246033 14414 log_verifier.cc:126] Checking tablet 80152844286545c68c7656ef9bbe8604
I20250110 03:07:30.256101 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 80152844286545c68c7656ef9bbe8604
I20250110 03:07:30.256151 14414 log_verifier.cc:126] Checking tablet 81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:30.266906 14414 log_verifier.cc:177] Verified matching terms for 26 ops in tablet 81b3bd116ad54a9fa10d2d0c829bb467
I20250110 03:07:30.266947 14414 log_verifier.cc:126] Checking tablet 87f8afb021a7407089c710f6dd588d03
I20250110 03:07:30.276839 14414 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 87f8afb021a7407089c710f6dd588d03
I20250110 03:07:30.276883 14414 log_verifier.cc:126] Checking tablet 9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:30.286471 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 9b16001c451b455f9eda42f3c98a2fdd
I20250110 03:07:30.286518 14414 log_verifier.cc:126] Checking tablet a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:30.296132 14414 log_verifier.cc:177] Verified matching terms for 8 ops in tablet a6000ace3863451db3fdd8d150cd0944
I20250110 03:07:30.296175 14414 log_verifier.cc:126] Checking tablet a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:30.306063 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet a790eedb619d47c6a3ba7eba194b70c1
I20250110 03:07:30.306118 14414 log_verifier.cc:126] Checking tablet abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:30.315879 14414 log_verifier.cc:177] Verified matching terms for 12 ops in tablet abfade13c24b4cebaf1ec204947154c0
I20250110 03:07:30.315922 14414 log_verifier.cc:126] Checking tablet b5064ab85103488f873fffef55fd78fd
I20250110 03:07:30.326390 14414 log_verifier.cc:177] Verified matching terms for 21 ops in tablet b5064ab85103488f873fffef55fd78fd
I20250110 03:07:30.326433 14414 log_verifier.cc:126] Checking tablet c5d0922741634918830ee2c163cbce24
I20250110 03:07:30.336369 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet c5d0922741634918830ee2c163cbce24
I20250110 03:07:30.336412 14414 log_verifier.cc:126] Checking tablet d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:30.346518 14414 log_verifier.cc:177] Verified matching terms for 15 ops in tablet d52b50e3c87b47bead3a25f7255d1c42
I20250110 03:07:30.346593 14414 log_verifier.cc:126] Checking tablet ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:30.356904 14414 log_verifier.cc:177] Verified matching terms for 16 ops in tablet ea70c2cc77384698aedb06bee2b639ce
I20250110 03:07:30.356948 14414 log_verifier.cc:126] Checking tablet ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:30.366466 14414 log_verifier.cc:177] Verified matching terms for 9 ops in tablet ecbd6afd612b4e86b94429633ac15082
I20250110 03:07:30.366509 14414 log_verifier.cc:126] Checking tablet f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:30.380175 14414 log_verifier.cc:177] Verified matching terms for 17 ops in tablet f1e7baec841a4b7e9cb933d786ec2552
I20250110 03:07:30.380224 14414 log_verifier.cc:126] Checking tablet fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:30.390476 14414 log_verifier.cc:177] Verified matching terms for 17 ops in tablet fcde555c3b364cf3b9b582ee3e273f3e
I20250110 03:07:30.390729 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21256
I20250110 03:07:30.397295 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21387
I20250110 03:07:30.403506 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21518
I20250110 03:07:30.409790 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21649
I20250110 03:07:30.415621 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 22889
I20250110 03:07:30.421191 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 23020
I20250110 03:07:30.426314 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 23151
I20250110 03:07:30.431418 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 23282
I20250110 03:07:30.436559 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 21165
2025-01-10T03:07:30Z chronyd exiting
[ OK ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1 (25345 ms)
[----------] 1 test from ConcurrentRebalancersTest (25345 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20250110 03:07:30.546398 14414 test_util.cc:274] Using random seed: 1697937114
I20250110 03:07:30.546911 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:07:30.546944 14414 ts_itest-base.cc:113] --------------
I20250110 03:07:30.546964 14414 ts_itest-base.cc:114] 4 tablet servers
I20250110 03:07:30.546983 14414 ts_itest-base.cc:115] 1 replicas per TS
I20250110 03:07:30.547000 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:07:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:07:30Z Disabled control of system clock
I20250110 03:07:30.554265 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:35615
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:35615
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20250110 03:07:30.627172 32204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:30.627319 32204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:30.627347 32204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:30.628527 32204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:30.628564 32204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:07:30.628578 32204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:30.628590 32204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:07:30.628602 32204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:07:30.629931 32204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:35615
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:30.630080 32204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:30.630257 32204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:30.633973 32212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:30.635020 32210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:30.636103 32209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:30.636999 32204 server_base.cc:1034] running on GCE node
I20250110 03:07:30.637249 32204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:30.637480 32204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:30.638604 32204 hybrid_clock.cc:648] HybridClock initialized: now 1736478450638592 us; error 32 us; skew 500 ppm
I20250110 03:07:30.645351 32204 webserver.cc:458] Webserver started at http://127.14.19.190:39567/ using document root <none> and password file <none>
I20250110 03:07:30.645504 32204 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:30.645543 32204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:30.645612 32204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:30.646401 32204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "70d7530389414043817bfe241cd99512"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.646694 32204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "70d7530389414043817bfe241cd99512"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.647794 32204 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:30.648388 32218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.648567 32204 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:30.648626 32204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "70d7530389414043817bfe241cd99512"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.648669 32204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:30.699996 32204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:30.700253 32204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:30.700340 32204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:30.703958 32204 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:35615
I20250110 03:07:30.703990 32270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:35615 every 8 connection(s)
I20250110 03:07:30.704279 32204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:07:30.704826 32271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:30.707319 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512: Bootstrap starting.
I20250110 03:07:30.707892 32271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:30.708125 32271 log.cc:826] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:30.708313 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 32204
I20250110 03:07:30.708418 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:07:30.708710 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512: No bootstrap required, opened a new log
I20250110 03:07:30.710433 32271 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d7530389414043817bfe241cd99512" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35615 } }
I20250110 03:07:30.710587 32271 raft_consensus.cc:383] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:30.710625 32271 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d7530389414043817bfe241cd99512, State: Initialized, Role: FOLLOWER
I20250110 03:07:30.710724 32271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d7530389414043817bfe241cd99512" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35615 } }
I20250110 03:07:30.710798 32271 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:30.710825 32271 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:30.710870 32271 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:30.711504 32271 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d7530389414043817bfe241cd99512" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35615 } }
I20250110 03:07:30.711607 32271 leader_election.cc:304] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70d7530389414043817bfe241cd99512; no voters:
I20250110 03:07:30.711731 32271 leader_election.cc:290] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:30.711809 32276 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:30.711928 32271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:07:30.711994 32276 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [term 1 LEADER]: Becoming Leader. State: Replica: 70d7530389414043817bfe241cd99512, State: Running, Role: LEADER
I20250110 03:07:30.712066 32276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d7530389414043817bfe241cd99512" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35615 } }
I20250110 03:07:30.712320 32276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d7530389414043817bfe241cd99512. Latest consensus state: current_term: 1 leader_uuid: "70d7530389414043817bfe241cd99512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d7530389414043817bfe241cd99512" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35615 } } }
I20250110 03:07:30.712414 32276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:30.712332 32277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70d7530389414043817bfe241cd99512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70d7530389414043817bfe241cd99512" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 35615 } } }
I20250110 03:07:30.712641 32282 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:07:30.713117 32282 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:07:30.713778 32277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:30.715070 32282 catalog_manager.cc:1348] Generated new cluster ID: f3651e970853424382c5256ccb4c5186
I20250110 03:07:30.715121 32282 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:07:30.727449 32282 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:07:30.728010 32282 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:07:30.733691 32282 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 70d7530389414043817bfe241cd99512: Generated new TSK 0
I20250110 03:07:30.733820 32282 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:07:30.740589 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:30.826140 32295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:30.826294 32295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:30.826321 32295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:30.827510 32295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:30.827550 32295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:30.827587 32295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:07:30.828913 32295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--never_fsync=true
--heap_profile_path=/tmp/kudu.32295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:30.829110 32295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:30.829324 32295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:30.833946 32300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:30.834924 32301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:30.835776 32303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:30.836539 32295 server_base.cc:1034] running on GCE node
I20250110 03:07:30.836694 32295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:30.836896 32295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:30.838032 32295 hybrid_clock.cc:648] HybridClock initialized: now 1736478450838016 us; error 35 us; skew 500 ppm
I20250110 03:07:30.839150 32295 webserver.cc:458] Webserver started at http://127.14.19.129:34703/ using document root <none> and password file <none>
I20250110 03:07:30.839334 32295 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:30.839380 32295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:30.839465 32295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:30.840289 32295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6815b8cbe23d41b6825372df8fbe3265"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.840569 32295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6815b8cbe23d41b6825372df8fbe3265"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.841701 32295 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:07:30.842334 32309 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.842519 32295 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.842624 32295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6815b8cbe23d41b6825372df8fbe3265"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.842687 32295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:30.879076 32295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:30.879331 32295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:30.879437 32295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:30.879626 32295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:30.879956 32295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:30.879997 32295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.880023 32295 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:30.880045 32295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.885658 32295 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:42139
I20250110 03:07:30.885735 32422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:42139 every 8 connection(s)
I20250110 03:07:30.885978 32295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:07:30.890704 32423 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35615
I20250110 03:07:30.890790 32423 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:30.890970 32423 heartbeater.cc:510] Master 127.14.19.190:35615 requested a full tablet report, sending...
I20250110 03:07:30.891381 32235 ts_manager.cc:194] Registered new tserver with Master: 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139)
I20250110 03:07:30.891948 32235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:46127
I20250110 03:07:30.895779 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 32295
I20250110 03:07:30.895852 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:07:30.896730 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:30.968252 32426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:30.968393 32426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:30.968412 32426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:30.969637 32426 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:30.969676 32426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:30.969707 32426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:07:30.971127 32426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--never_fsync=true
--heap_profile_path=/tmp/kudu.32426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:30.971323 32426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:30.971522 32426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:30.976115 32434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:30.976791 32432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:30.977154 32431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:30.978181 32426 server_base.cc:1034] running on GCE node
I20250110 03:07:30.978329 32426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:30.978514 32426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:30.979672 32426 hybrid_clock.cc:648] HybridClock initialized: now 1736478450979657 us; error 35 us; skew 500 ppm
I20250110 03:07:30.980722 32426 webserver.cc:458] Webserver started at http://127.14.19.130:37971/ using document root <none> and password file <none>
I20250110 03:07:30.980911 32426 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:30.980953 32426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:30.981040 32426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:30.981818 32426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0d2e3d701cb94a50ab78739e79a1765d"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.982081 32426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0d2e3d701cb94a50ab78739e79a1765d"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.983206 32426 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:07:30.983793 32440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.983935 32426 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:30.984001 32426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0d2e3d701cb94a50ab78739e79a1765d"
format_stamp: "Formatted at 2025-01-10 03:07:30 on dist-test-slave-nrvn"
I20250110 03:07:30.984055 32426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:30.993671 32426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:30.993841 32426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:30.993937 32426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:30.994082 32426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:30.994319 32426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:30.994350 32426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.994379 32426 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:30.994407 32426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:30.999660 32426 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:41045
I20250110 03:07:30.999713 32553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:41045 every 8 connection(s)
I20250110 03:07:30.999970 32426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:07:31.000329 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 32426
I20250110 03:07:31.000401 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:07:31.001530 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:31.004663 32554 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35615
I20250110 03:07:31.004762 32554 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:31.004971 32554 heartbeater.cc:510] Master 127.14.19.190:35615 requested a full tablet report, sending...
I20250110 03:07:31.005326 32235 ts_manager.cc:194] Registered new tserver with Master: 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:31.005671 32235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:45119
W20250110 03:07:31.076838 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:31.076992 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:31.077023 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:31.078233 32557 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:31.078275 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:31.078323 32557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:07:31.079720 32557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--never_fsync=true
--heap_profile_path=/tmp/kudu.32557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:31.079944 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:31.080150 32557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:31.083985 32562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:31.084985 32565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:31.085741 32563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:31.086715 32557 server_base.cc:1034] running on GCE node
I20250110 03:07:31.086871 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:31.087056 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:31.088183 32557 hybrid_clock.cc:648] HybridClock initialized: now 1736478451088174 us; error 27 us; skew 500 ppm
I20250110 03:07:31.089232 32557 webserver.cc:458] Webserver started at http://127.14.19.131:36191/ using document root <none> and password file <none>
I20250110 03:07:31.089397 32557 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:31.089437 32557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:31.089504 32557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:31.090292 32557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e4b12c76c1714aa0a7ff6c53f9496b43"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:31.090580 32557 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e4b12c76c1714aa0a7ff6c53f9496b43"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:31.091709 32557 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:31.092300 32571 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:31.092448 32557 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:31.092518 32557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e4b12c76c1714aa0a7ff6c53f9496b43"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:31.092574 32557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:31.103534 32557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:31.103734 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:31.103832 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:31.103986 32557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:31.104244 32557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:31.104274 32557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:31.104306 32557 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:31.104337 32557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:31.109705 32557 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:35111
I20250110 03:07:31.109743 32684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:35111 every 8 connection(s)
I20250110 03:07:31.110016 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:07:31.114696 32685 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35615
I20250110 03:07:31.114779 32685 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:31.114976 32685 heartbeater.cc:510] Master 127.14.19.190:35615 requested a full tablet report, sending...
I20250110 03:07:31.115473 32235 ts_manager.cc:194] Registered new tserver with Master: e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131:35111)
I20250110 03:07:31.115779 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 32557
I20250110 03:07:31.115859 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:07:31.115936 32235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:41989
I20250110 03:07:31.116866 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:31.189566 32688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:31.189705 32688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:31.189743 32688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:31.190984 32688 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:31.191022 32688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:31.191043 32688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:31.192389 32688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:35615
--never_fsync=true
--heap_profile_path=/tmp/kudu.32688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:31.192552 32688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:31.192718 32688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:31.196398 32694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:31.197387 32693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:31.198166 32696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:31.199070 32688 server_base.cc:1034] running on GCE node
I20250110 03:07:31.199224 32688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:31.199400 32688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:31.200523 32688 hybrid_clock.cc:648] HybridClock initialized: now 1736478451200508 us; error 33 us; skew 500 ppm
I20250110 03:07:31.201717 32688 webserver.cc:458] Webserver started at http://127.14.19.132:35319/ using document root <none> and password file <none>
I20250110 03:07:31.201901 32688 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:31.201946 32688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:31.202029 32688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:31.202875 32688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ab9e18e6490f4943b0abfdf89d971108"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:31.203138 32688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ab9e18e6490f4943b0abfdf89d971108"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:31.204211 32688 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:31.204749 32702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:31.204955 32688 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:31.205040 32688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ab9e18e6490f4943b0abfdf89d971108"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:31.205096 32688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:31.212141 32688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:31.212328 32688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:31.212455 32688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:31.212622 32688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:31.212904 32688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:31.212939 32688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:31.213017 32688 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:31.213049 32688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:31.218236 32688 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:38847
I20250110 03:07:31.218268 347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:38847 every 8 connection(s)
I20250110 03:07:31.218504 32688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:31.220564 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 32688
I20250110 03:07:31.220644 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:07:31.222820 348 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35615
I20250110 03:07:31.222913 348 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:31.223048 348 heartbeater.cc:510] Master 127.14.19.190:35615 requested a full tablet report, sending...
I20250110 03:07:31.223369 32235 ts_manager.cc:194] Registered new tserver with Master: ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132:38847)
I20250110 03:07:31.223722 32235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:50611
I20250110 03:07:31.232223 14414 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250110 03:07:31.237141 32235 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32794:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250110 03:07:31.243192 32619 tablet_service.cc:1467] Processing CreateTablet for tablet 78298fadb7eb464caf47cbf44a9ccd61 (DEFAULT_TABLE table=TestTable [id=30b7e5a3008d4409939d5509963b4f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:31.243485 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78298fadb7eb464caf47cbf44a9ccd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:31.245451 361 tablet_bootstrap.cc:492] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap starting.
I20250110 03:07:31.246125 361 tablet_bootstrap.cc:654] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:31.246414 361 log.cc:826] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:31.246996 361 tablet_bootstrap.cc:492] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: No bootstrap required, opened a new log
I20250110 03:07:31.247043 361 ts_tablet_manager.cc:1397] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:31.248709 361 raft_consensus.cc:357] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:31.248812 361 raft_consensus.cc:383] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:31.248832 361 raft_consensus.cc:738] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Initialized, Role: FOLLOWER
I20250110 03:07:31.248910 361 consensus_queue.cc:260] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:31.248970 361 raft_consensus.cc:397] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:31.249003 361 raft_consensus.cc:491] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:31.249047 361 raft_consensus.cc:3054] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:31.249538 361 raft_consensus.cc:513] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:31.249634 361 leader_election.cc:304] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b12c76c1714aa0a7ff6c53f9496b43; no voters:
I20250110 03:07:31.249799 361 leader_election.cc:290] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:31.249845 363 raft_consensus.cc:2798] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:31.249974 361 ts_tablet_manager.cc:1428] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250110 03:07:31.249990 363 raft_consensus.cc:695] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEADER]: Becoming Leader. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Running, Role: LEADER
I20250110 03:07:31.250017 32685 heartbeater.cc:502] Master 127.14.19.190:35615 was elected leader, sending a full tablet report...
I20250110 03:07:31.250066 363 consensus_queue.cc:237] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:31.250931 32235 catalog_manager.cc:5526] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:31.253011 14414 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250110 03:07:31.253389 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6815b8cbe23d41b6825372df8fbe3265 to finish bootstrapping
I20250110 03:07:31.256559 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0d2e3d701cb94a50ab78739e79a1765d to finish bootstrapping
I20250110 03:07:31.259696 14414 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e4b12c76c1714aa0a7ff6c53f9496b43 to finish bootstrapping
I20250110 03:07:31.262368 14414 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ab9e18e6490f4943b0abfdf89d971108 to finish bootstrapping
I20250110 03:07:31.265807 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 32557
I20250110 03:07:31.271265 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 32688
I20250110 03:07:31.892680 32423 heartbeater.cc:502] Master 127.14.19.190:35615 was elected leader, sending a full tablet report...
I20250110 03:07:32.006498 32554 heartbeater.cc:502] Master 127.14.19.190:35615 was elected leader, sending a full tablet report...
I20250110 03:07:35.025952 32235 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32794:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:35.031105 32488 tablet_service.cc:1467] Processing CreateTablet for tablet 02fe4e06947b45aea91c9fda946d0707 (DEFAULT_TABLE table=rebalance_test_table_0 [id=11318b52b2414a80ac4a44c7127629ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.031144 32356 tablet_service.cc:1467] Processing CreateTablet for tablet 9bffd88289a84709b7a93ff6404c9ee8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=11318b52b2414a80ac4a44c7127629ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.031144 32357 tablet_service.cc:1467] Processing CreateTablet for tablet 62872e590b72414a81a79e9936027ae8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=11318b52b2414a80ac4a44c7127629ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.031512 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bffd88289a84709b7a93ff6404c9ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.031514 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe4e06947b45aea91c9fda946d0707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.031627 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62872e590b72414a81a79e9936027ae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.034240 381 tablet_bootstrap.cc:492] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d: Bootstrap starting.
I20250110 03:07:35.034271 380 tablet_bootstrap.cc:492] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Bootstrap starting.
I20250110 03:07:35.034765 381 tablet_bootstrap.cc:654] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.034792 380 tablet_bootstrap.cc:654] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.035007 380 log.cc:826] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:35.035121 381 log.cc:826] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:35.035590 380 tablet_bootstrap.cc:492] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: No bootstrap required, opened a new log
I20250110 03:07:35.035658 380 ts_tablet_manager.cc:1397] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:35.035671 381 tablet_bootstrap.cc:492] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d: No bootstrap required, opened a new log
I20250110 03:07:35.035717 381 ts_tablet_manager.cc:1397] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:35.037807 380 raft_consensus.cc:357] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.037815 381 raft_consensus.cc:357] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.037930 381 raft_consensus.cc:383] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.037930 380 raft_consensus.cc:383] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.037958 381 raft_consensus.cc:738] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.037958 380 raft_consensus.cc:738] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.038053 380 consensus_queue.cc:260] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.038079 381 consensus_queue.cc:260] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.038120 380 raft_consensus.cc:397] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.038132 381 raft_consensus.cc:397] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.038161 381 raft_consensus.cc:491] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.038161 380 raft_consensus.cc:491] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.038205 380 raft_consensus.cc:3054] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.038205 381 raft_consensus.cc:3054] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.038815 380 raft_consensus.cc:513] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.038815 381 raft_consensus.cc:513] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.038918 381 leader_election.cc:304] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d; no voters:
I20250110 03:07:35.038918 380 leader_election.cc:304] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265; no voters:
I20250110 03:07:35.039063 380 leader_election.cc:290] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.039072 381 leader_election.cc:290] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.039140 385 raft_consensus.cc:2798] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.039261 380 ts_tablet_manager.cc:1428] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250110 03:07:35.039261 381 ts_tablet_manager.cc:1428] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250110 03:07:35.039320 385 raft_consensus.cc:695] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Leader. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:35.039364 380 tablet_bootstrap.cc:492] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265: Bootstrap starting.
I20250110 03:07:35.039402 385 consensus_queue.cc:237] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.039568 387 raft_consensus.cc:2798] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.039705 387 raft_consensus.cc:695] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Leader. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:35.039785 387 consensus_queue.cc:237] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.039997 32235 catalog_manager.cc:5526] T 02fe4e06947b45aea91c9fda946d0707 P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130). New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.040659 380 tablet_bootstrap.cc:654] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.040900 32235 catalog_manager.cc:5526] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.041481 380 tablet_bootstrap.cc:492] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265: No bootstrap required, opened a new log
I20250110 03:07:35.041528 380 ts_tablet_manager.cc:1397] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:35.041649 380 raft_consensus.cc:357] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.041687 380 raft_consensus.cc:383] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.041699 380 raft_consensus.cc:738] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.041743 380 consensus_queue.cc:260] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.041777 380 raft_consensus.cc:397] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.041802 380 raft_consensus.cc:491] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.041828 380 raft_consensus.cc:3054] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.042217 380 raft_consensus.cc:513] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.042264 380 leader_election.cc:304] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265; no voters:
I20250110 03:07:35.042323 380 leader_election.cc:290] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.042361 387 raft_consensus.cc:2798] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.042407 380 ts_tablet_manager.cc:1428] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:35.042434 387 raft_consensus.cc:695] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Leader. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:35.042480 387 consensus_queue.cc:237] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.042932 32235 catalog_manager.cc:5526] T 62872e590b72414a81a79e9936027ae8 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.143193 32235 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32794:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:35.145483 32357 tablet_service.cc:1467] Processing CreateTablet for tablet 80982ad268db4d9a9660ddfbda0a1e98 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b038446bd56b45bd8b51ee3db7d53a5c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.145454 32488 tablet_service.cc:1467] Processing CreateTablet for tablet babadeef5e7a40f89435f2803603c9ab (DEFAULT_TABLE table=rebalance_test_table_1 [id=b038446bd56b45bd8b51ee3db7d53a5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.145479 32487 tablet_service.cc:1467] Processing CreateTablet for tablet c5ed9433a60e42d098d28eaa6a016c1e (DEFAULT_TABLE table=rebalance_test_table_1 [id=b038446bd56b45bd8b51ee3db7d53a5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.145607 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80982ad268db4d9a9660ddfbda0a1e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.145629 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5ed9433a60e42d098d28eaa6a016c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.145720 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet babadeef5e7a40f89435f2803603c9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.146608 380 tablet_bootstrap.cc:492] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265: Bootstrap starting.
I20250110 03:07:35.147042 381 tablet_bootstrap.cc:492] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Bootstrap starting.
I20250110 03:07:35.147235 380 tablet_bootstrap.cc:654] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.147400 381 tablet_bootstrap.cc:654] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.147760 380 tablet_bootstrap.cc:492] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265: No bootstrap required, opened a new log
I20250110 03:07:35.147816 380 ts_tablet_manager.cc:1397] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:35.147964 380 raft_consensus.cc:357] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.147991 381 tablet_bootstrap.cc:492] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: No bootstrap required, opened a new log
I20250110 03:07:35.148018 380 raft_consensus.cc:383] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.148026 381 ts_tablet_manager.cc:1397] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.148042 380 raft_consensus.cc:738] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.148097 380 consensus_queue.cc:260] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.148140 380 raft_consensus.cc:397] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.148164 380 raft_consensus.cc:491] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.148159 381 raft_consensus.cc:357] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.148213 380 raft_consensus.cc:3054] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.148226 381 raft_consensus.cc:383] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.148247 381 raft_consensus.cc:738] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.148311 381 consensus_queue.cc:260] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.148361 381 raft_consensus.cc:397] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.148379 381 raft_consensus.cc:491] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.148407 381 raft_consensus.cc:3054] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.148756 380 raft_consensus.cc:513] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.148813 380 leader_election.cc:304] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265; no voters:
I20250110 03:07:35.148824 381 raft_consensus.cc:513] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.148852 380 leader_election.cc:290] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.148893 381 leader_election.cc:304] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d; no voters:
I20250110 03:07:35.148963 381 leader_election.cc:290] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.148969 380 ts_tablet_manager.cc:1428] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 03:07:35.149060 390 raft_consensus.cc:2798] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.149052 381 ts_tablet_manager.cc:1428] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.149053 386 raft_consensus.cc:2798] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.149101 390 raft_consensus.cc:695] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Leader. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:35.149116 386 raft_consensus.cc:695] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Leader. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:35.149142 390 consensus_queue.cc:237] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.149161 386 consensus_queue.cc:237] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.149564 32235 catalog_manager.cc:5526] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130). New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.149123 381 tablet_bootstrap.cc:492] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Bootstrap starting.
I20250110 03:07:35.149910 32234 catalog_manager.cc:5526] T 80982ad268db4d9a9660ddfbda0a1e98 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.150302 381 tablet_bootstrap.cc:654] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.151042 381 tablet_bootstrap.cc:492] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: No bootstrap required, opened a new log
I20250110 03:07:35.151082 381 ts_tablet_manager.cc:1397] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:35.151185 381 raft_consensus.cc:357] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.151221 381 raft_consensus.cc:383] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.151242 381 raft_consensus.cc:738] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.151278 381 consensus_queue.cc:260] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.151345 381 raft_consensus.cc:397] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.151364 381 raft_consensus.cc:491] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.151384 381 raft_consensus.cc:3054] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.151768 381 raft_consensus.cc:513] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.151814 381 leader_election.cc:304] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d; no voters:
I20250110 03:07:35.151842 381 leader_election.cc:290] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.151880 385 raft_consensus.cc:2798] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.151933 381 ts_tablet_manager.cc:1428] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:35.151978 385 raft_consensus.cc:695] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Leader. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:35.152172 385 consensus_queue.cc:237] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.152607 32234 catalog_manager.cc:5526] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130). New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.255698 32234 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32794:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:35.257102 32488 tablet_service.cc:1467] Processing CreateTablet for tablet 7d92b7bc1b364c8683ed8f52a1445895 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0edad7631fac480bbc04b5146385dfe8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.257110 32356 tablet_service.cc:1467] Processing CreateTablet for tablet 4470d26030b444c9a8ca93562433c0d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0edad7631fac480bbc04b5146385dfe8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.257099 32357 tablet_service.cc:1467] Processing CreateTablet for tablet 38850c90b0d94738bfe01d6c60bb473f (DEFAULT_TABLE table=rebalance_test_table_2 [id=0edad7631fac480bbc04b5146385dfe8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.257236 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d92b7bc1b364c8683ed8f52a1445895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.257252 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4470d26030b444c9a8ca93562433c0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.257375 32357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38850c90b0d94738bfe01d6c60bb473f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.258160 381 tablet_bootstrap.cc:492] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d: Bootstrap starting.
I20250110 03:07:35.258450 380 tablet_bootstrap.cc:492] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265: Bootstrap starting.
I20250110 03:07:35.258746 381 tablet_bootstrap.cc:654] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.258924 380 tablet_bootstrap.cc:654] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.259331 381 tablet_bootstrap.cc:492] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d: No bootstrap required, opened a new log
I20250110 03:07:35.259373 381 ts_tablet_manager.cc:1397] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:35.259488 381 raft_consensus.cc:357] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.259531 381 raft_consensus.cc:383] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.259545 381 raft_consensus.cc:738] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.259577 381 consensus_queue.cc:260] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.259626 381 raft_consensus.cc:397] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.259642 381 raft_consensus.cc:491] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.259663 381 raft_consensus.cc:3054] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.260093 381 raft_consensus.cc:513] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.260159 381 leader_election.cc:304] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d; no voters:
I20250110 03:07:35.260200 381 leader_election.cc:290] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.260239 385 raft_consensus.cc:2798] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.260279 381 ts_tablet_manager.cc:1428] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:35.260298 385 raft_consensus.cc:695] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Leader. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:35.260334 385 consensus_queue.cc:237] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.260614 380 tablet_bootstrap.cc:492] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265: No bootstrap required, opened a new log
I20250110 03:07:35.260656 380 ts_tablet_manager.cc:1397] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:35.260733 32234 catalog_manager.cc:5526] T 7d92b7bc1b364c8683ed8f52a1445895 P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130). New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.260816 380 raft_consensus.cc:357] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.260877 380 raft_consensus.cc:383] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.260905 380 raft_consensus.cc:738] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.260951 380 consensus_queue.cc:260] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.260995 380 raft_consensus.cc:397] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.261029 380 raft_consensus.cc:491] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.261061 380 raft_consensus.cc:3054] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.261546 380 raft_consensus.cc:513] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.261602 380 leader_election.cc:304] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265; no voters:
I20250110 03:07:35.261634 380 leader_election.cc:290] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.261709 384 raft_consensus.cc:2798] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.261709 380 ts_tablet_manager.cc:1428] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.261801 384 raft_consensus.cc:695] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Leader. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:35.261822 380 tablet_bootstrap.cc:492] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Bootstrap starting.
I20250110 03:07:35.261848 384 consensus_queue.cc:237] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.262251 380 tablet_bootstrap.cc:654] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.262367 32234 catalog_manager.cc:5526] T 4470d26030b444c9a8ca93562433c0d9 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.262881 380 tablet_bootstrap.cc:492] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: No bootstrap required, opened a new log
I20250110 03:07:35.262925 380 ts_tablet_manager.cc:1397] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.263139 380 raft_consensus.cc:357] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.263182 380 raft_consensus.cc:383] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.263195 380 raft_consensus.cc:738] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.263231 380 consensus_queue.cc:260] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.263273 380 raft_consensus.cc:397] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.263299 380 raft_consensus.cc:491] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.263334 380 raft_consensus.cc:3054] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.263788 380 raft_consensus.cc:513] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.263839 380 leader_election.cc:304] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265; no voters:
I20250110 03:07:35.263948 380 leader_election.cc:290] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.263972 387 raft_consensus.cc:2798] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.264026 387 raft_consensus.cc:695] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Leader. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:35.264055 380 ts_tablet_manager.cc:1428] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.264076 387 consensus_queue.cc:237] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.264508 32234 catalog_manager.cc:5526] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.362536 32234 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32794:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250110 03:07:35.363917 32488 tablet_service.cc:1467] Processing CreateTablet for tablet a0888f8ca6284065be9f7db277e6e414 (DEFAULT_TABLE table=rebalance_test_table_3 [id=deb2a5543c7e45e1936e33a0774c4a20]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.363929 32356 tablet_service.cc:1467] Processing CreateTablet for tablet fab48f4443a54481b79a36b27d399383 (DEFAULT_TABLE table=rebalance_test_table_3 [id=deb2a5543c7e45e1936e33a0774c4a20]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.363953 32487 tablet_service.cc:1467] Processing CreateTablet for tablet 8d8d1cd3a6324b4e9cc287d52f646010 (DEFAULT_TABLE table=rebalance_test_table_3 [id=deb2a5543c7e45e1936e33a0774c4a20]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:35.364051 32488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0888f8ca6284065be9f7db277e6e414. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.364051 32356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab48f4443a54481b79a36b27d399383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.364154 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8d1cd3a6324b4e9cc287d52f646010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:35.364912 381 tablet_bootstrap.cc:492] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d: Bootstrap starting.
I20250110 03:07:35.365307 380 tablet_bootstrap.cc:492] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265: Bootstrap starting.
I20250110 03:07:35.365352 381 tablet_bootstrap.cc:654] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.365739 380 tablet_bootstrap.cc:654] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.366047 381 tablet_bootstrap.cc:492] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d: No bootstrap required, opened a new log
I20250110 03:07:35.366092 381 ts_tablet_manager.cc:1397] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:35.366250 381 raft_consensus.cc:357] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.366298 381 raft_consensus.cc:383] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.366312 381 raft_consensus.cc:738] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.366340 380 tablet_bootstrap.cc:492] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265: No bootstrap required, opened a new log
I20250110 03:07:35.366348 381 consensus_queue.cc:260] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.366371 380 ts_tablet_manager.cc:1397] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.366386 381 raft_consensus.cc:397] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.366405 381 raft_consensus.cc:491] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.366423 381 raft_consensus.cc:3054] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.366478 380 raft_consensus.cc:357] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.366513 380 raft_consensus.cc:383] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.366526 380 raft_consensus.cc:738] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.366581 380 consensus_queue.cc:260] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.366627 380 raft_consensus.cc:397] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.366650 380 raft_consensus.cc:491] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.366678 380 raft_consensus.cc:3054] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.366936 381 raft_consensus.cc:513] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.366992 381 leader_election.cc:304] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d; no voters:
I20250110 03:07:35.367024 381 leader_election.cc:290] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.367130 381 ts_tablet_manager.cc:1428] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.367195 381 tablet_bootstrap.cc:492] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Bootstrap starting.
I20250110 03:07:35.367231 380 raft_consensus.cc:513] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.367300 380 leader_election.cc:304] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265; no voters:
I20250110 03:07:35.367350 380 leader_election.cc:290] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.367422 380 ts_tablet_manager.cc:1428] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.367514 388 raft_consensus.cc:2798] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.367568 388 raft_consensus.cc:695] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Leader. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:35.367574 381 tablet_bootstrap.cc:654] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:35.367630 388 consensus_queue.cc:237] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.367805 387 raft_consensus.cc:2798] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.367853 387 raft_consensus.cc:695] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Leader. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:35.367890 387 consensus_queue.cc:237] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } }
I20250110 03:07:35.368270 32235 catalog_manager.cc:5526] T fab48f4443a54481b79a36b27d399383 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.368494 32234 catalog_manager.cc:5526] T a0888f8ca6284065be9f7db277e6e414 P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130). New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.369068 381 tablet_bootstrap.cc:492] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: No bootstrap required, opened a new log
I20250110 03:07:35.369107 381 ts_tablet_manager.cc:1397] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:35.369210 381 raft_consensus.cc:357] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.369246 381 raft_consensus.cc:383] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:35.369258 381 raft_consensus.cc:738] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.369287 381 consensus_queue.cc:260] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.369330 381 raft_consensus.cc:397] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.369345 381 raft_consensus.cc:491] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.369364 381 raft_consensus.cc:3054] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:35.369742 381 raft_consensus.cc:513] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.369791 381 leader_election.cc:304] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d; no voters:
I20250110 03:07:35.369818 381 leader_election.cc:290] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:35.369855 386 raft_consensus.cc:2798] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:35.369889 381 ts_tablet_manager.cc:1428] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:35.369911 386 raft_consensus.cc:695] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Leader. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:35.369943 386 consensus_queue.cc:237] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } }
I20250110 03:07:35.370342 32233 catalog_manager.cc:5526] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130). New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.471360 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:35111
--local_ip_for_outbound_sockets=127.14.19.131
--tserver_master_addrs=127.14.19.190:35615
--webserver_port=36191
--webserver_interface=127.14.19.131
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:35.559105 486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:35.559247 486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:35.559265 486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:35.560488 486 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:35.560528 486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:35.560552 486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:07:35.561902 486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:35111
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=36191
--tserver_master_addrs=127.14.19.190:35615
--never_fsync=true
--heap_profile_path=/tmp/kudu.486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:35.562119 486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:35.562338 486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:35.566296 491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:35.567272 494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:35.568368 492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:35.569144 486 server_base.cc:1034] running on GCE node
I20250110 03:07:35.569278 486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:35.569475 486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:35.570592 486 hybrid_clock.cc:648] HybridClock initialized: now 1736478455570578 us; error 33 us; skew 500 ppm
I20250110 03:07:35.571652 486 webserver.cc:458] Webserver started at http://127.14.19.131:36191/ using document root <none> and password file <none>
I20250110 03:07:35.571846 486 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:35.571916 486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:35.572963 486 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:35.573498 500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.573647 486 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.573706 486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e4b12c76c1714aa0a7ff6c53f9496b43"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:35.573969 486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:35.585724 486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:35.585906 486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:35.585994 486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:35.586174 486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:35.586508 507 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250110 03:07:35.587217 486 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250110 03:07:35.587257 486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:35.587278 486 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250110 03:07:35.587702 486 ts_tablet_manager.cc:610] Registered 1 tablets
I20250110 03:07:35.587730 486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.587800 507 tablet_bootstrap.cc:492] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap starting.
I20250110 03:07:35.594528 486 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:35111
I20250110 03:07:35.594623 614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:35111 every 8 connection(s)
I20250110 03:07:35.594878 486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:07:35.595712 507 log.cc:826] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:35.596634 507 tablet_bootstrap.cc:492] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:35.596913 507 tablet_bootstrap.cc:492] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap complete.
I20250110 03:07:35.597864 507 ts_tablet_manager.cc:1397] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250110 03:07:35.599157 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 486
I20250110 03:07:35.599433 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:38847
--local_ip_for_outbound_sockets=127.14.19.132
--tserver_master_addrs=127.14.19.190:35615
--webserver_port=35319
--webserver_interface=127.14.19.132
--builtin_ntp_servers=127.14.19.148:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:35.599346 507 raft_consensus.cc:357] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:35.599543 507 raft_consensus.cc:738] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Initialized, Role: FOLLOWER
I20250110 03:07:35.599660 507 consensus_queue.cc:260] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:35.599718 507 raft_consensus.cc:397] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:35.599746 507 raft_consensus.cc:491] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:35.599790 507 raft_consensus.cc:3054] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:35.600704 507 raft_consensus.cc:513] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:35.600816 507 leader_election.cc:304] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4b12c76c1714aa0a7ff6c53f9496b43; no voters:
I20250110 03:07:35.600955 615 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35615
I20250110 03:07:35.600957 507 leader_election.cc:290] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250110 03:07:35.601080 615 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:35.601008 620 raft_consensus.cc:2798] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:35.601263 507 ts_tablet_manager.cc:1428] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
I20250110 03:07:35.601303 615 heartbeater.cc:510] Master 127.14.19.190:35615 requested a full tablet report, sending...
I20250110 03:07:35.601294 620 raft_consensus.cc:695] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 LEADER]: Becoming Leader. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Running, Role: LEADER
I20250110 03:07:35.601420 620 consensus_queue.cc:237] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } }
I20250110 03:07:35.602144 32233 ts_manager.cc:194] Re-registered known tserver with Master: e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131:35111)
I20250110 03:07:35.602203 32233 catalog_manager.cc:5526] T 78298fadb7eb464caf47cbf44a9ccd61 P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:35.603116 32233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:33927
I20250110 03:07:35.603724 615 heartbeater.cc:502] Master 127.14.19.190:35615 was elected leader, sending a full tablet report...
W20250110 03:07:35.674221 619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:35.674365 619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:35.674384 619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:35.675562 619 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:35.675602 619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:35.675624 619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:35.676895 619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:38847
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=35319
--tserver_master_addrs=127.14.19.190:35615
--never_fsync=true
--heap_profile_path=/tmp/kudu.619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:35.677042 619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:35.677225 619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:35.681033 630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:35.681932 632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:35.682914 629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:35.683693 619 server_base.cc:1034] running on GCE node
I20250110 03:07:35.683836 619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:35.684007 619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:35.685130 619 hybrid_clock.cc:648] HybridClock initialized: now 1736478455685113 us; error 31 us; skew 500 ppm
I20250110 03:07:35.690853 619 webserver.cc:458] Webserver started at http://127.14.19.132:35319/ using document root <none> and password file <none>
I20250110 03:07:35.691047 619 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:35.691118 619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:35.692268 619 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:35.692756 638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.692903 619 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.692972 619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ab9e18e6490f4943b0abfdf89d971108"
format_stamp: "Formatted at 2025-01-10 03:07:31 on dist-test-slave-nrvn"
I20250110 03:07:35.693234 619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:35.706647 619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:35.706851 619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:35.706952 619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:35.707129 619 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:35.707393 619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:35.707423 619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.707454 619 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:35.707479 619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:35.712714 619 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:38847
I20250110 03:07:35.712767 751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:38847 every 8 connection(s)
I20250110 03:07:35.713032 619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:35.714160 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 619
I20250110 03:07:35.717574 752 heartbeater.cc:346] Connected to a master server at 127.14.19.190:35615
I20250110 03:07:35.717653 752 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:35.717809 752 heartbeater.cc:510] Master 127.14.19.190:35615 requested a full tablet report, sending...
I20250110 03:07:35.718147 32233 ts_manager.cc:194] Re-registered known tserver with Master: ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132:38847)
I20250110 03:07:35.718497 32233 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:54739
I20250110 03:07:35.824023 32356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:35.824401 32487 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:35.826265 686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:35.828718 549 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:35.868289 32487 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:35.868877 32356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:35.869094 549 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:35.872582 686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:35.923571 32356 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:35.923945 686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:35.926589 549 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:35.928989 32487 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:35.941581 32377 consensus_queue.cc:237] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:35.941980 866 raft_consensus.cc:2949] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } } }
I20250110 03:07:35.943286 32234 catalog_manager.cc:5526] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: config changed from index -1 to 3, NON_VOTER ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:35.945194 32311 consensus_peers.cc:487] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 -> Peer ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132:38847): Couldn't send request to peer ab9e18e6490f4943b0abfdf89d971108. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38850c90b0d94738bfe01d6c60bb473f. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:35.948570 32508 consensus_queue.cc:237] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } }
I20250110 03:07:35.948927 385 raft_consensus.cc:2949] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } } }
I20250110 03:07:35.950088 32234 catalog_manager.cc:5526] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: config changed from index -1 to 2, NON_VOTER e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) added. New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:35.952845 32441 consensus_peers.cc:487] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d -> Peer e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131:35111): Couldn't send request to peer e4b12c76c1714aa0a7ff6c53f9496b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5ed9433a60e42d098d28eaa6a016c1e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:35.954385 32377 consensus_queue.cc:237] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } }
I20250110 03:07:35.954713 867 raft_consensus.cc:2949] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } } }
I20250110 03:07:35.955603 32234 catalog_manager.cc:5526] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: config changed from index -1 to 2, NON_VOTER e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) added. New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:35.957847 32310 consensus_peers.cc:487] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 -> Peer e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131:35111): Couldn't send request to peer e4b12c76c1714aa0a7ff6c53f9496b43. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bffd88289a84709b7a93ff6404c9ee8. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:35.960055 32508 consensus_queue.cc:237] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:35.960361 874 raft_consensus.cc:2949] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } } }
I20250110 03:07:35.961441 32234 catalog_manager.cc:5526] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: config changed from index -1 to 2, NON_VOTER ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:35.964038 32442 consensus_peers.cc:487] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d -> Peer ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132:38847): Couldn't send request to peer ab9e18e6490f4943b0abfdf89d971108. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d8d1cd3a6324b4e9cc287d52f646010. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:35.965601 32508 consensus_queue.cc:237] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:35.965827 385 raft_consensus.cc:2949] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } } }
W20250110 03:07:35.966015 32442 consensus_peers.cc:487] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d -> Peer ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132:38847): Couldn't send request to peer ab9e18e6490f4943b0abfdf89d971108. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: babadeef5e7a40f89435f2803603c9ab. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:35.966789 32234 catalog_manager.cc:5526] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: config changed from index -1 to 3, NON_VOTER ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:36.335881 942 ts_tablet_manager.cc:927] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: Initiating tablet copy from peer 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139)
I20250110 03:07:36.336081 942 tablet_copy_client.cc:323] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42139
I20250110 03:07:36.338639 32397 tablet_copy_service.cc:140] P 6815b8cbe23d41b6825372df8fbe3265: Received BeginTabletCopySession request for tablet 9bffd88289a84709b7a93ff6404c9ee8 from peer e4b12c76c1714aa0a7ff6c53f9496b43 ({username='slave'} at 127.14.19.131:47809)
I20250110 03:07:36.338711 32397 tablet_copy_service.cc:161] P 6815b8cbe23d41b6825372df8fbe3265: Beginning new tablet copy session on tablet 9bffd88289a84709b7a93ff6404c9ee8 from peer e4b12c76c1714aa0a7ff6c53f9496b43 at {username='slave'} at 127.14.19.131:47809: session id = e4b12c76c1714aa0a7ff6c53f9496b43-9bffd88289a84709b7a93ff6404c9ee8
I20250110 03:07:36.339248 32397 tablet_copy_source_session.cc:215] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:36.339846 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bffd88289a84709b7a93ff6404c9ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:36.341075 942 tablet_copy_client.cc:806] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:36.341215 942 tablet_copy_client.cc:670] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:36.341933 942 tablet_copy_client.cc:538] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:36.342468 942 tablet_bootstrap.cc:492] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap starting.
I20250110 03:07:36.343688 942 tablet_bootstrap.cc:492] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:36.343828 942 tablet_bootstrap.cc:492] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap complete.
I20250110 03:07:36.343895 942 ts_tablet_manager.cc:1397] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:36.344033 942 raft_consensus.cc:357] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } }
I20250110 03:07:36.344084 942 raft_consensus.cc:738] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Initialized, Role: LEARNER
I20250110 03:07:36.344112 942 consensus_queue.cc:260] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } }
I20250110 03:07:36.344244 942 ts_tablet_manager.cc:1428] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:36.344445 32397 tablet_copy_service.cc:342] P 6815b8cbe23d41b6825372df8fbe3265: Request end of tablet copy session e4b12c76c1714aa0a7ff6c53f9496b43-9bffd88289a84709b7a93ff6404c9ee8 received from {username='slave'} at 127.14.19.131:47809
I20250110 03:07:36.344498 32397 tablet_copy_service.cc:434] P 6815b8cbe23d41b6825372df8fbe3265: ending tablet copy session e4b12c76c1714aa0a7ff6c53f9496b43-9bffd88289a84709b7a93ff6404c9ee8 on tablet 9bffd88289a84709b7a93ff6404c9ee8 with peer e4b12c76c1714aa0a7ff6c53f9496b43
I20250110 03:07:36.382798 947 ts_tablet_manager.cc:927] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Initiating tablet copy from peer 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:36.382974 947 tablet_copy_client.cc:323] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:41045
I20250110 03:07:36.385469 32528 tablet_copy_service.cc:140] P 0d2e3d701cb94a50ab78739e79a1765d: Received BeginTabletCopySession request for tablet babadeef5e7a40f89435f2803603c9ab from peer ab9e18e6490f4943b0abfdf89d971108 ({username='slave'} at 127.14.19.132:33933)
I20250110 03:07:36.385532 32528 tablet_copy_service.cc:161] P 0d2e3d701cb94a50ab78739e79a1765d: Beginning new tablet copy session on tablet babadeef5e7a40f89435f2803603c9ab from peer ab9e18e6490f4943b0abfdf89d971108 at {username='slave'} at 127.14.19.132:33933: session id = ab9e18e6490f4943b0abfdf89d971108-babadeef5e7a40f89435f2803603c9ab
I20250110 03:07:36.386018 32528 tablet_copy_source_session.cc:215] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:36.386750 947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet babadeef5e7a40f89435f2803603c9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:36.388119 950 ts_tablet_manager.cc:927] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: Initiating tablet copy from peer 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139)
I20250110 03:07:36.388264 950 tablet_copy_client.cc:323] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:42139
I20250110 03:07:36.389187 947 tablet_copy_client.cc:806] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:36.389331 947 tablet_copy_client.cc:670] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:36.390100 947 tablet_copy_client.cc:538] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:36.390761 32397 tablet_copy_service.cc:140] P 6815b8cbe23d41b6825372df8fbe3265: Received BeginTabletCopySession request for tablet 38850c90b0d94738bfe01d6c60bb473f from peer ab9e18e6490f4943b0abfdf89d971108 ({username='slave'} at 127.14.19.132:45343)
I20250110 03:07:36.390825 32397 tablet_copy_service.cc:161] P 6815b8cbe23d41b6825372df8fbe3265: Beginning new tablet copy session on tablet 38850c90b0d94738bfe01d6c60bb473f from peer ab9e18e6490f4943b0abfdf89d971108 at {username='slave'} at 127.14.19.132:45343: session id = ab9e18e6490f4943b0abfdf89d971108-38850c90b0d94738bfe01d6c60bb473f
I20250110 03:07:36.390833 947 tablet_bootstrap.cc:492] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Bootstrap starting.
I20250110 03:07:36.391397 32397 tablet_copy_source_session.cc:215] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:36.391709 950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38850c90b0d94738bfe01d6c60bb473f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:36.392206 947 log.cc:826] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:36.392575 950 tablet_copy_client.cc:806] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:36.392685 950 tablet_copy_client.cc:670] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:36.393229 950 tablet_copy_client.cc:538] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:36.393867 947 tablet_bootstrap.cc:492] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:36.393934 950 tablet_bootstrap.cc:492] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: Bootstrap starting.
I20250110 03:07:36.394074 947 tablet_bootstrap.cc:492] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Bootstrap complete.
I20250110 03:07:36.394177 947 ts_tablet_manager.cc:1397] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:36.395053 947 raft_consensus.cc:357] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:36.395169 947 raft_consensus.cc:738] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab9e18e6490f4943b0abfdf89d971108, State: Initialized, Role: LEARNER
I20250110 03:07:36.395231 947 consensus_queue.cc:260] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:36.395239 950 tablet_bootstrap.cc:492] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:36.395427 947 ts_tablet_manager.cc:1428] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250110 03:07:36.395467 752 heartbeater.cc:502] Master 127.14.19.190:35615 was elected leader, sending a full tablet report...
I20250110 03:07:36.395654 950 tablet_bootstrap.cc:492] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: Bootstrap complete.
I20250110 03:07:36.395694 32528 tablet_copy_service.cc:342] P 0d2e3d701cb94a50ab78739e79a1765d: Request end of tablet copy session ab9e18e6490f4943b0abfdf89d971108-babadeef5e7a40f89435f2803603c9ab received from {username='slave'} at 127.14.19.132:33933
I20250110 03:07:36.395748 32528 tablet_copy_service.cc:434] P 0d2e3d701cb94a50ab78739e79a1765d: ending tablet copy session ab9e18e6490f4943b0abfdf89d971108-babadeef5e7a40f89435f2803603c9ab on tablet babadeef5e7a40f89435f2803603c9ab with peer ab9e18e6490f4943b0abfdf89d971108
I20250110 03:07:36.395748 950 ts_tablet_manager.cc:1397] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250110 03:07:36.395903 950 raft_consensus.cc:357] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:36.395952 950 raft_consensus.cc:738] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab9e18e6490f4943b0abfdf89d971108, State: Initialized, Role: LEARNER
I20250110 03:07:36.395987 950 consensus_queue.cc:260] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:36.396063 950 ts_tablet_manager.cc:1428] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:36.396381 32397 tablet_copy_service.cc:342] P 6815b8cbe23d41b6825372df8fbe3265: Request end of tablet copy session ab9e18e6490f4943b0abfdf89d971108-38850c90b0d94738bfe01d6c60bb473f received from {username='slave'} at 127.14.19.132:45343
I20250110 03:07:36.396435 32397 tablet_copy_service.cc:434] P 6815b8cbe23d41b6825372df8fbe3265: ending tablet copy session ab9e18e6490f4943b0abfdf89d971108-38850c90b0d94738bfe01d6c60bb473f on tablet 38850c90b0d94738bfe01d6c60bb473f with peer ab9e18e6490f4943b0abfdf89d971108
I20250110 03:07:36.550508 950 ts_tablet_manager.cc:927] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: Initiating tablet copy from peer 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:36.550729 950 tablet_copy_client.cc:323] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:41045
I20250110 03:07:36.551015 32528 tablet_copy_service.cc:140] P 0d2e3d701cb94a50ab78739e79a1765d: Received BeginTabletCopySession request for tablet 8d8d1cd3a6324b4e9cc287d52f646010 from peer ab9e18e6490f4943b0abfdf89d971108 ({username='slave'} at 127.14.19.132:33933)
I20250110 03:07:36.551086 32528 tablet_copy_service.cc:161] P 0d2e3d701cb94a50ab78739e79a1765d: Beginning new tablet copy session on tablet 8d8d1cd3a6324b4e9cc287d52f646010 from peer ab9e18e6490f4943b0abfdf89d971108 at {username='slave'} at 127.14.19.132:33933: session id = ab9e18e6490f4943b0abfdf89d971108-8d8d1cd3a6324b4e9cc287d52f646010
I20250110 03:07:36.551477 32528 tablet_copy_source_session.cc:215] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:36.551829 950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d8d1cd3a6324b4e9cc287d52f646010. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:36.552747 950 tablet_copy_client.cc:806] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:36.552855 950 tablet_copy_client.cc:670] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:36.553416 950 tablet_copy_client.cc:538] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:36.553926 950 tablet_bootstrap.cc:492] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: Bootstrap starting.
I20250110 03:07:36.555024 950 tablet_bootstrap.cc:492] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:36.555187 950 tablet_bootstrap.cc:492] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: Bootstrap complete.
I20250110 03:07:36.555269 950 ts_tablet_manager.cc:1397] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:36.555402 950 raft_consensus.cc:357] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:36.555462 950 raft_consensus.cc:738] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab9e18e6490f4943b0abfdf89d971108, State: Initialized, Role: LEARNER
I20250110 03:07:36.555495 950 consensus_queue.cc:260] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: true } }
I20250110 03:07:36.555588 950 ts_tablet_manager.cc:1428] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:36.555785 32528 tablet_copy_service.cc:342] P 0d2e3d701cb94a50ab78739e79a1765d: Request end of tablet copy session ab9e18e6490f4943b0abfdf89d971108-8d8d1cd3a6324b4e9cc287d52f646010 received from {username='slave'} at 127.14.19.132:33933
I20250110 03:07:36.555840 32528 tablet_copy_service.cc:434] P 0d2e3d701cb94a50ab78739e79a1765d: ending tablet copy session ab9e18e6490f4943b0abfdf89d971108-8d8d1cd3a6324b4e9cc287d52f646010 on tablet 8d8d1cd3a6324b4e9cc287d52f646010 with peer ab9e18e6490f4943b0abfdf89d971108
I20250110 03:07:36.556103 942 ts_tablet_manager.cc:927] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: Initiating tablet copy from peer 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:36.556241 942 tablet_copy_client.cc:323] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:41045
I20250110 03:07:36.558434 32528 tablet_copy_service.cc:140] P 0d2e3d701cb94a50ab78739e79a1765d: Received BeginTabletCopySession request for tablet c5ed9433a60e42d098d28eaa6a016c1e from peer e4b12c76c1714aa0a7ff6c53f9496b43 ({username='slave'} at 127.14.19.131:40073)
I20250110 03:07:36.558481 32528 tablet_copy_service.cc:161] P 0d2e3d701cb94a50ab78739e79a1765d: Beginning new tablet copy session on tablet c5ed9433a60e42d098d28eaa6a016c1e from peer e4b12c76c1714aa0a7ff6c53f9496b43 at {username='slave'} at 127.14.19.131:40073: session id = e4b12c76c1714aa0a7ff6c53f9496b43-c5ed9433a60e42d098d28eaa6a016c1e
I20250110 03:07:36.558836 32528 tablet_copy_source_session.cc:215] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:36.559119 942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5ed9433a60e42d098d28eaa6a016c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:36.559917 942 tablet_copy_client.cc:806] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:36.560014 942 tablet_copy_client.cc:670] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:36.560539 942 tablet_copy_client.cc:538] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:36.561025 942 tablet_bootstrap.cc:492] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap starting.
I20250110 03:07:36.562086 942 tablet_bootstrap.cc:492] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:36.562239 942 tablet_bootstrap.cc:492] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: Bootstrap complete.
I20250110 03:07:36.562317 942 ts_tablet_manager.cc:1397] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:36.562469 942 raft_consensus.cc:357] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } }
I20250110 03:07:36.562518 942 raft_consensus.cc:738] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Initialized, Role: LEARNER
I20250110 03:07:36.562563 942 consensus_queue.cc:260] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: NON_VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: true } }
I20250110 03:07:36.562655 942 ts_tablet_manager.cc:1428] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:36.562845 32528 tablet_copy_service.cc:342] P 0d2e3d701cb94a50ab78739e79a1765d: Request end of tablet copy session e4b12c76c1714aa0a7ff6c53f9496b43-c5ed9433a60e42d098d28eaa6a016c1e received from {username='slave'} at 127.14.19.131:40073
I20250110 03:07:36.562896 32528 tablet_copy_service.cc:434] P 0d2e3d701cb94a50ab78739e79a1765d: ending tablet copy session e4b12c76c1714aa0a7ff6c53f9496b43-c5ed9433a60e42d098d28eaa6a016c1e on tablet c5ed9433a60e42d098d28eaa6a016c1e with peer e4b12c76c1714aa0a7ff6c53f9496b43
I20250110 03:07:36.861933 569 raft_consensus.cc:1212] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250110 03:07:36.904512 706 raft_consensus.cc:1212] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20250110 03:07:36.928095 706 raft_consensus.cc:1212] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20250110 03:07:37.022182 706 raft_consensus.cc:1212] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250110 03:07:37.106295 569 raft_consensus.cc:1212] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250110 03:07:37.302587 1034 raft_consensus.cc:1059] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: attempting to promote NON_VOTER e4b12c76c1714aa0a7ff6c53f9496b43 to VOTER
I20250110 03:07:37.302817 1034 consensus_queue.cc:237] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:37.303336 569 raft_consensus.cc:1270] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Refusing update from remote peer 6815b8cbe23d41b6825372df8fbe3265: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:37.303577 866 consensus_queue.cc:1035] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 03:07:37.304432 1034 raft_consensus.cc:2949] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:37.304601 569 raft_consensus.cc:2949] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:37.305796 32234 catalog_manager.cc:5526] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 reported cstate change: config changed from index 2 to 3, e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:37.336565 866 raft_consensus.cc:1059] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: attempting to promote NON_VOTER ab9e18e6490f4943b0abfdf89d971108 to VOTER
I20250110 03:07:37.336704 866 consensus_queue.cc:237] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:37.337122 706 raft_consensus.cc:1270] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Refusing update from remote peer 6815b8cbe23d41b6825372df8fbe3265: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:37.337338 866 consensus_queue.cc:1035] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:37.338270 1035 raft_consensus.cc:2949] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.338415 706 raft_consensus.cc:2949] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.339689 32232 catalog_manager.cc:5526] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: config changed from index 3 to 4, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6815b8cbe23d41b6825372df8fbe3265" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.454991 891 raft_consensus.cc:1059] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: attempting to promote NON_VOTER ab9e18e6490f4943b0abfdf89d971108 to VOTER
I20250110 03:07:37.455216 891 consensus_queue.cc:237] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:37.455649 706 raft_consensus.cc:1270] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Refusing update from remote peer 0d2e3d701cb94a50ab78739e79a1765d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:37.455869 891 consensus_queue.cc:1035] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:37.456629 1036 raft_consensus.cc:2949] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.456804 706 raft_consensus.cc:2949] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.457784 32232 catalog_manager.cc:5526] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: config changed from index 3 to 4, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.499811 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "babadeef5e7a40f89435f2803603c9ab"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50106
I20250110 03:07:37.499924 32508 raft_consensus.cc:604] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.510057 32377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bffd88289a84709b7a93ff6404c9ee8"
dest_uuid: "6815b8cbe23d41b6825372df8fbe3265"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57176
I20250110 03:07:37.510152 32377 raft_consensus.cc:604] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.516671 32377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38850c90b0d94738bfe01d6c60bb473f"
dest_uuid: "6815b8cbe23d41b6825372df8fbe3265"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57190
I20250110 03:07:37.516760 32377 raft_consensus.cc:604] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.542135 891 raft_consensus.cc:1059] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: attempting to promote NON_VOTER ab9e18e6490f4943b0abfdf89d971108 to VOTER
I20250110 03:07:37.542284 891 consensus_queue.cc:237] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:37.542762 706 raft_consensus.cc:1270] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 LEARNER]: Refusing update from remote peer 0d2e3d701cb94a50ab78739e79a1765d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:37.542968 891 consensus_queue.cc:1035] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 03:07:37.543781 1036 raft_consensus.cc:2949] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.543993 706 raft_consensus.cc:2949] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:37.545028 32234 catalog_manager.cc:5526] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d reported cstate change: config changed from index 2 to 3, ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:37.650066 1036 raft_consensus.cc:1059] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: attempting to promote NON_VOTER e4b12c76c1714aa0a7ff6c53f9496b43 to VOTER
I20250110 03:07:37.650244 1036 consensus_queue.cc:237] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:37.650707 569 raft_consensus.cc:1270] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 LEARNER]: Refusing update from remote peer 0d2e3d701cb94a50ab78739e79a1765d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:37.650907 1036 consensus_queue.cc:1035] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 03:07:37.651621 891 raft_consensus.cc:2949] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:37.651752 569 raft_consensus.cc:2949] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:37.652804 32234 catalog_manager.cc:5526] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: config changed from index 2 to 3, e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d2e3d701cb94a50ab78739e79a1765d" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:37.727442 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "babadeef5e7a40f89435f2803603c9ab"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50138
I20250110 03:07:37.727521 32508 raft_consensus.cc:604] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.734596 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d8d1cd3a6324b4e9cc287d52f646010"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50154
I20250110 03:07:37.734673 32508 raft_consensus.cc:604] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.737267 1035 raft_consensus.cc:988] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: : Instructing follower e4b12c76c1714aa0a7ff6c53f9496b43 to start an election
I20250110 03:07:37.737357 1035 raft_consensus.cc:1076] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Signalling peer e4b12c76c1714aa0a7ff6c53f9496b43 to start an election
I20250110 03:07:37.737581 569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bffd88289a84709b7a93ff6404c9ee8"
dest_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43"
from {username='slave'} at 127.14.19.129:38185
I20250110 03:07:37.737663 569 raft_consensus.cc:491] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:37.737702 569 raft_consensus.cc:3054] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:37.738238 569 raft_consensus.cc:513] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:37.738656 32377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bffd88289a84709b7a93ff6404c9ee8" candidate_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6815b8cbe23d41b6825372df8fbe3265"
I20250110 03:07:37.738757 32377 raft_consensus.cc:3049] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:37.738466 569 leader_election.cc:290] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 2 election: Requested vote from peers 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139)
I20250110 03:07:37.738798 32377 raft_consensus.cc:738] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:37.738862 32377 consensus_queue.cc:260] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:37.738927 32377 raft_consensus.cc:3054] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:37.739564 32377 raft_consensus.cc:2463] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4b12c76c1714aa0a7ff6c53f9496b43 in term 2.
I20250110 03:07:37.739745 504 leader_election.cc:304] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265, e4b12c76c1714aa0a7ff6c53f9496b43; no voters:
I20250110 03:07:37.739885 946 raft_consensus.cc:2798] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:37.739989 946 raft_consensus.cc:695] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 LEADER]: Becoming Leader. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Running, Role: LEADER
I20250110 03:07:37.740065 946 consensus_queue.cc:237] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:37.740527 32232 catalog_manager.cc:5526] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: term changed from 1 to 2, leader changed from 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129) to e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131). New cstate: current_term: 2 leader_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250110 03:07:37.744573 32219 catalog_manager.cc:5056] ChangeConfig:REMOVE_PEER RPC for tablet 9bffd88289a84709b7a93ff6404c9ee8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131:35111) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250110 03:07:37.746739 32376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38850c90b0d94738bfe01d6c60bb473f"
dest_uuid: "6815b8cbe23d41b6825372df8fbe3265"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49708
I20250110 03:07:37.746809 32376 raft_consensus.cc:604] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.753568 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5ed9433a60e42d098d28eaa6a016c1e"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59852
I20250110 03:07:37.753633 32508 raft_consensus.cc:604] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.776244 1035 raft_consensus.cc:988] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: : Instructing follower ab9e18e6490f4943b0abfdf89d971108 to start an election
I20250110 03:07:37.776314 1035 raft_consensus.cc:1076] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Signalling peer ab9e18e6490f4943b0abfdf89d971108 to start an election
I20250110 03:07:37.776496 706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38850c90b0d94738bfe01d6c60bb473f"
dest_uuid: "ab9e18e6490f4943b0abfdf89d971108"
from {username='slave'} at 127.14.19.129:60297
I20250110 03:07:37.776585 706 raft_consensus.cc:491] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:37.776625 706 raft_consensus.cc:3054] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:37.777143 706 raft_consensus.cc:513] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:37.777349 706 leader_election.cc:290] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [CANDIDATE]: Term 2 election: Requested vote from peers 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139)
I20250110 03:07:37.777534 32376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38850c90b0d94738bfe01d6c60bb473f" candidate_uuid: "ab9e18e6490f4943b0abfdf89d971108" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6815b8cbe23d41b6825372df8fbe3265"
I20250110 03:07:37.777613 32376 raft_consensus.cc:3049] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:37.777637 32376 raft_consensus.cc:738] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6815b8cbe23d41b6825372df8fbe3265, State: Running, Role: LEADER
I20250110 03:07:37.777709 32376 consensus_queue.cc:260] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:37.777796 32376 raft_consensus.cc:3054] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:37.778345 32376 raft_consensus.cc:2463] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9e18e6490f4943b0abfdf89d971108 in term 2.
I20250110 03:07:37.778513 642 leader_election.cc:304] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6815b8cbe23d41b6825372df8fbe3265, ab9e18e6490f4943b0abfdf89d971108; no voters:
I20250110 03:07:37.778674 954 raft_consensus.cc:2798] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:37.778796 954 raft_consensus.cc:695] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 2 LEADER]: Becoming Leader. State: Replica: ab9e18e6490f4943b0abfdf89d971108, State: Running, Role: LEADER
I20250110 03:07:37.778959 954 consensus_queue.cc:237] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:37.779524 32232 catalog_manager.cc:5526] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: term changed from 1 to 2, leader changed from 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129) to ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132). New cstate: current_term: 2 leader_uuid: "ab9e18e6490f4943b0abfdf89d971108" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:37.961355 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "babadeef5e7a40f89435f2803603c9ab"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59872
I20250110 03:07:37.961431 32508 raft_consensus.cc:604] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.968164 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d8d1cd3a6324b4e9cc287d52f646010"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59894
I20250110 03:07:37.968228 32508 raft_consensus.cc:604] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:37.981704 32508 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5ed9433a60e42d098d28eaa6a016c1e"
dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59900
I20250110 03:07:37.981766 32508 raft_consensus.cc:604] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:38.003932 1037 raft_consensus.cc:988] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: : Instructing follower ab9e18e6490f4943b0abfdf89d971108 to start an election
I20250110 03:07:38.003989 1037 raft_consensus.cc:1076] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Signalling peer ab9e18e6490f4943b0abfdf89d971108 to start an election
I20250110 03:07:38.004173 706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "babadeef5e7a40f89435f2803603c9ab"
dest_uuid: "ab9e18e6490f4943b0abfdf89d971108"
from {username='slave'} at 127.14.19.130:45551
I20250110 03:07:38.004241 706 raft_consensus.cc:491] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:38.004262 706 raft_consensus.cc:3054] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:38.004755 706 raft_consensus.cc:513] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.004864 706 leader_election.cc:290] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [CANDIDATE]: Term 2 election: Requested vote from peers 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:38.005033 32508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babadeef5e7a40f89435f2803603c9ab" candidate_uuid: "ab9e18e6490f4943b0abfdf89d971108" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
I20250110 03:07:38.005111 32508 raft_consensus.cc:3049] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:38.005141 32508 raft_consensus.cc:738] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:38.005193 32508 consensus_queue.cc:260] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.005262 32508 raft_consensus.cc:3054] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:38.005861 32508 raft_consensus.cc:2463] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9e18e6490f4943b0abfdf89d971108 in term 2.
I20250110 03:07:38.006027 639 leader_election.cc:304] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d, ab9e18e6490f4943b0abfdf89d971108; no voters:
I20250110 03:07:38.006134 954 raft_consensus.cc:2798] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:38.006196 954 raft_consensus.cc:695] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 2 LEADER]: Becoming Leader. State: Replica: ab9e18e6490f4943b0abfdf89d971108, State: Running, Role: LEADER
I20250110 03:07:38.006247 954 consensus_queue.cc:237] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.006680 32232 catalog_manager.cc:5526] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: term changed from 1 to 2, leader changed from 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) to ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132). New cstate: current_term: 2 leader_uuid: "ab9e18e6490f4943b0abfdf89d971108" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.116367 32376 raft_consensus.cc:1270] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Refusing update from remote peer e4b12c76c1714aa0a7ff6c53f9496b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 03:07:38.116672 946 consensus_queue.cc:1035] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:38.120018 1037 raft_consensus.cc:988] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: : Instructing follower ab9e18e6490f4943b0abfdf89d971108 to start an election
I20250110 03:07:38.120090 1037 raft_consensus.cc:1076] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Signalling peer ab9e18e6490f4943b0abfdf89d971108 to start an election
I20250110 03:07:38.120292 706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d8d1cd3a6324b4e9cc287d52f646010"
dest_uuid: "ab9e18e6490f4943b0abfdf89d971108"
from {username='slave'} at 127.14.19.130:45551
I20250110 03:07:38.120363 706 raft_consensus.cc:491] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:38.120395 706 raft_consensus.cc:3054] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:38.120951 706 raft_consensus.cc:513] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.121060 706 leader_election.cc:290] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [CANDIDATE]: Term 2 election: Requested vote from peers 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:38.121224 32508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d8d1cd3a6324b4e9cc287d52f646010" candidate_uuid: "ab9e18e6490f4943b0abfdf89d971108" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
I20250110 03:07:38.121289 32508 raft_consensus.cc:3049] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:38.121315 32508 raft_consensus.cc:738] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:38.121388 32508 consensus_queue.cc:260] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.121452 32508 raft_consensus.cc:3054] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:38.121981 32508 raft_consensus.cc:2463] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9e18e6490f4943b0abfdf89d971108 in term 2.
I20250110 03:07:38.122125 639 leader_election.cc:304] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d, ab9e18e6490f4943b0abfdf89d971108; no voters:
I20250110 03:07:38.122224 954 raft_consensus.cc:2798] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:38.122268 954 raft_consensus.cc:695] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 2 LEADER]: Becoming Leader. State: Replica: ab9e18e6490f4943b0abfdf89d971108, State: Running, Role: LEADER
I20250110 03:07:38.122304 954 consensus_queue.cc:237] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.122726 32234 catalog_manager.cc:5526] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: term changed from 1 to 2, leader changed from 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) to ab9e18e6490f4943b0abfdf89d971108 (127.14.19.132). New cstate: current_term: 2 leader_uuid: "ab9e18e6490f4943b0abfdf89d971108" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.181182 1037 raft_consensus.cc:988] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: : Instructing follower e4b12c76c1714aa0a7ff6c53f9496b43 to start an election
I20250110 03:07:38.181270 1037 raft_consensus.cc:1076] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Signalling peer e4b12c76c1714aa0a7ff6c53f9496b43 to start an election
I20250110 03:07:38.181548 569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5ed9433a60e42d098d28eaa6a016c1e"
dest_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43"
from {username='slave'} at 127.14.19.130:59607
I20250110 03:07:38.181628 569 raft_consensus.cc:491] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:38.181650 569 raft_consensus.cc:3054] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:38.182220 569 raft_consensus.cc:513] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:38.182351 569 leader_election.cc:290] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 2 election: Requested vote from peers 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045)
I20250110 03:07:38.182535 32508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ed9433a60e42d098d28eaa6a016c1e" candidate_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0d2e3d701cb94a50ab78739e79a1765d"
I20250110 03:07:38.182631 32508 raft_consensus.cc:3049] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:38.182650 32508 raft_consensus.cc:738] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d2e3d701cb94a50ab78739e79a1765d, State: Running, Role: LEADER
I20250110 03:07:38.182708 32508 consensus_queue.cc:260] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:38.182780 32508 raft_consensus.cc:3054] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:38.183456 32508 raft_consensus.cc:2463] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4b12c76c1714aa0a7ff6c53f9496b43 in term 2.
I20250110 03:07:38.183662 501 leader_election.cc:304] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d2e3d701cb94a50ab78739e79a1765d, e4b12c76c1714aa0a7ff6c53f9496b43; no voters:
I20250110 03:07:38.183779 946 raft_consensus.cc:2798] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:38.183843 946 raft_consensus.cc:695] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 LEADER]: Becoming Leader. State: Replica: e4b12c76c1714aa0a7ff6c53f9496b43, State: Running, Role: LEADER
I20250110 03:07:38.183908 946 consensus_queue.cc:237] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:38.184369 32234 catalog_manager.cc:5526] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: term changed from 1 to 2, leader changed from 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) to e4b12c76c1714aa0a7ff6c53f9496b43 (127.14.19.131). New cstate: current_term: 2 leader_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.197360 32376 raft_consensus.cc:1270] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Refusing update from remote peer ab9e18e6490f4943b0abfdf89d971108: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250110 03:07:38.197634 954 consensus_queue.cc:1035] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815b8cbe23d41b6825372df8fbe3265" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 42139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:38.412824 569 consensus_queue.cc:237] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:38.413156 946 raft_consensus.cc:2949] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:38.414115 32219 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 9bffd88289a84709b7a93ff6404c9ee8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250110 03:07:38.414252 32234 catalog_manager.cc:5526] T 9bffd88289a84709b7a93ff6404c9ee8 P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: config changed from index 3 to 5, VOTER 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.415200 32356 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bffd88289a84709b7a93ff6404c9ee8 with delete_type TABLET_DATA_TOMBSTONED (TS 6815b8cbe23d41b6825372df8fbe3265 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56992
I20250110 03:07:38.415395 1275 tablet_replica.cc:331] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: stopping tablet replica
I20250110 03:07:38.415498 1275 raft_consensus.cc:2238] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:38.415598 1275 raft_consensus.cc:2267] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:38.415958 1275 ts_tablet_manager.cc:1905] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:38.417342 1275 ts_tablet_manager.cc:1918] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250110 03:07:38.417400 1275 log.cc:1198] T 9bffd88289a84709b7a93ff6404c9ee8 P 6815b8cbe23d41b6825372df8fbe3265: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/9bffd88289a84709b7a93ff6404c9ee8
I20250110 03:07:38.417724 32222 catalog_manager.cc:4872] TS 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139): tablet 9bffd88289a84709b7a93ff6404c9ee8 (table rebalance_test_table_0 [id=11318b52b2414a80ac4a44c7127629ed]) successfully deleted
I20250110 03:07:38.464306 705 consensus_queue.cc:237] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.464597 1190 raft_consensus.cc:2949] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:38.465396 32220 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 38850c90b0d94738bfe01d6c60bb473f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250110 03:07:38.465598 32234 catalog_manager.cc:5526] T 38850c90b0d94738bfe01d6c60bb473f P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: config changed from index 4 to 6, VOTER 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "ab9e18e6490f4943b0abfdf89d971108" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.466347 32356 tablet_service.cc:1514] Processing DeleteTablet for tablet 38850c90b0d94738bfe01d6c60bb473f with delete_type TABLET_DATA_TOMBSTONED (TS 6815b8cbe23d41b6825372df8fbe3265 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56992
I20250110 03:07:38.466460 1275 tablet_replica.cc:331] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: stopping tablet replica
I20250110 03:07:38.466522 1275 raft_consensus.cc:2238] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:38.466588 1275 raft_consensus.cc:2267] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:38.466828 1275 ts_tablet_manager.cc:1905] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:38.467983 1275 ts_tablet_manager.cc:1918] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20250110 03:07:38.468034 1275 log.cc:1198] T 38850c90b0d94738bfe01d6c60bb473f P 6815b8cbe23d41b6825372df8fbe3265: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/38850c90b0d94738bfe01d6c60bb473f
I20250110 03:07:38.468346 32222 catalog_manager.cc:4872] TS 6815b8cbe23d41b6825372df8fbe3265 (127.14.19.129:42139): tablet 38850c90b0d94738bfe01d6c60bb473f (table rebalance_test_table_2 [id=0edad7631fac480bbc04b5146385dfe8]) successfully deleted
I20250110 03:07:38.535506 32508 raft_consensus.cc:1270] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Refusing update from remote peer ab9e18e6490f4943b0abfdf89d971108: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250110 03:07:38.535796 954 consensus_queue.cc:1035] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:38.611438 32508 raft_consensus.cc:1270] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Refusing update from remote peer ab9e18e6490f4943b0abfdf89d971108: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 03:07:38.611707 954 consensus_queue.cc:1035] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:38.624429 705 consensus_queue.cc:237] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.624668 954 raft_consensus.cc:2949] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:38.625547 32220 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet babadeef5e7a40f89435f2803603c9ab with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250110 03:07:38.625721 32232 catalog_manager.cc:5526] T babadeef5e7a40f89435f2803603c9ab P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: config changed from index 4 to 6, VOTER 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "ab9e18e6490f4943b0abfdf89d971108" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.626505 32487 tablet_service.cc:1514] Processing DeleteTablet for tablet babadeef5e7a40f89435f2803603c9ab with delete_type TABLET_DATA_TOMBSTONED (TS 0d2e3d701cb94a50ab78739e79a1765d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49832
I20250110 03:07:38.626754 1309 tablet_replica.cc:331] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: stopping tablet replica
I20250110 03:07:38.626849 1309 raft_consensus.cc:2238] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:38.626907 1309 raft_consensus.cc:2267] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:38.627187 1309 ts_tablet_manager.cc:1905] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:38.628386 1309 ts_tablet_manager.cc:1918] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20250110 03:07:38.628443 1309 log.cc:1198] T babadeef5e7a40f89435f2803603c9ab P 0d2e3d701cb94a50ab78739e79a1765d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/babadeef5e7a40f89435f2803603c9ab
I20250110 03:07:38.628800 32219 catalog_manager.cc:4872] TS 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045): tablet babadeef5e7a40f89435f2803603c9ab (table rebalance_test_table_1 [id=b038446bd56b45bd8b51ee3db7d53a5c]) successfully deleted
I20250110 03:07:38.692327 32508 raft_consensus.cc:1270] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Refusing update from remote peer e4b12c76c1714aa0a7ff6c53f9496b43: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 03:07:38.692569 1167 consensus_queue.cc:1035] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d2e3d701cb94a50ab78739e79a1765d" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 41045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:38.711882 705 consensus_queue.cc:237] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } }
I20250110 03:07:38.712147 1190 raft_consensus.cc:2949] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } } }
I20250110 03:07:38.713099 32220 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8d8d1cd3a6324b4e9cc287d52f646010 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250110 03:07:38.713282 32234 catalog_manager.cc:5526] T 8d8d1cd3a6324b4e9cc287d52f646010 P ab9e18e6490f4943b0abfdf89d971108 reported cstate change: config changed from index 3 to 5, VOTER 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "ab9e18e6490f4943b0abfdf89d971108" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab9e18e6490f4943b0abfdf89d971108" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 38847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.714109 32487 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d8d1cd3a6324b4e9cc287d52f646010 with delete_type TABLET_DATA_TOMBSTONED (TS 0d2e3d701cb94a50ab78739e79a1765d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49832
I20250110 03:07:38.714236 1309 tablet_replica.cc:331] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: stopping tablet replica
I20250110 03:07:38.714309 1309 raft_consensus.cc:2238] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:38.714352 1309 raft_consensus.cc:2267] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:38.714563 1309 ts_tablet_manager.cc:1905] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:38.715646 1309 ts_tablet_manager.cc:1918] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250110 03:07:38.715699 1309 log.cc:1198] T 8d8d1cd3a6324b4e9cc287d52f646010 P 0d2e3d701cb94a50ab78739e79a1765d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d8d1cd3a6324b4e9cc287d52f646010
I20250110 03:07:38.716008 32219 catalog_manager.cc:4872] TS 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045): tablet 8d8d1cd3a6324b4e9cc287d52f646010 (table rebalance_test_table_3 [id=deb2a5543c7e45e1936e33a0774c4a20]) successfully deleted
I20250110 03:07:38.824709 569 consensus_queue.cc:237] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } }
I20250110 03:07:38.825023 946 raft_consensus.cc:2949] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } } }
I20250110 03:07:38.825810 32219 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c5ed9433a60e42d098d28eaa6a016c1e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250110 03:07:38.826031 32234 catalog_manager.cc:5526] T c5ed9433a60e42d098d28eaa6a016c1e P e4b12c76c1714aa0a7ff6c53f9496b43 reported cstate change: config changed from index 3 to 5, VOTER 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e4b12c76c1714aa0a7ff6c53f9496b43" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 35111 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:38.826856 32487 tablet_service.cc:1514] Processing DeleteTablet for tablet c5ed9433a60e42d098d28eaa6a016c1e with delete_type TABLET_DATA_TOMBSTONED (TS 0d2e3d701cb94a50ab78739e79a1765d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49832
I20250110 03:07:38.826964 1309 tablet_replica.cc:331] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: stopping tablet replica
I20250110 03:07:38.827020 1309 raft_consensus.cc:2238] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:38.827056 1309 raft_consensus.cc:2267] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:38.827281 1309 ts_tablet_manager.cc:1905] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:38.828410 1309 ts_tablet_manager.cc:1918] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250110 03:07:38.828459 1309 log.cc:1198] T c5ed9433a60e42d098d28eaa6a016c1e P 0d2e3d701cb94a50ab78739e79a1765d: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/c5ed9433a60e42d098d28eaa6a016c1e
I20250110 03:07:38.828733 32219 catalog_manager.cc:4872] TS 0d2e3d701cb94a50ab78739e79a1765d (127.14.19.130:41045): tablet c5ed9433a60e42d098d28eaa6a016c1e (table rebalance_test_table_1 [id=b038446bd56b45bd8b51ee3db7d53a5c]) successfully deleted
I20250110 03:07:38.892068 549 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.892403 686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.892793 32356 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:38.897677 32487 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.928344 32487 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.928623 549 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.929826 32356 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:38.933182 686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.971282 32487 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.972044 32356 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:38.972326 686 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:38.972510 549 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
70d7530389414043817bfe241cd99512 | 127.14.19.190:35615 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:42289 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0d2e3d701cb94a50ab78739e79a1765d | 127.14.19.130:41045 | HEALTHY | <none> | 3 | 0
6815b8cbe23d41b6825372df8fbe3265 | 127.14.19.129:42139 | HEALTHY | <none> | 4 | 0
ab9e18e6490f4943b0abfdf89d971108 | 127.14.19.132:38847 | HEALTHY | <none> | 3 | 0
e4b12c76c1714aa0a7ff6c53f9496b43 | 127.14.19.131:35111 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.14.19.129 | experimental | 127.14.19.129:42139
local_ip_for_outbound_sockets | 127.14.19.130 | experimental | 127.14.19.130:41045
local_ip_for_outbound_sockets | 127.14.19.131 | experimental | 127.14.19.131:35111
local_ip_for_outbound_sockets | 127.14.19.132 | experimental | 127.14.19.132:38847
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.14.19.129:42139
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.14.19.130:41045
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.14.19.131:35111
server_dump_info_path | /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.14.19.132:38847
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.14.19.148:42289 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 03:07:38.982188 14414 log_verifier.cc:126] Checking tablet 02fe4e06947b45aea91c9fda946d0707
I20250110 03:07:38.988293 14414 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 02fe4e06947b45aea91c9fda946d0707
I20250110 03:07:38.988337 14414 log_verifier.cc:126] Checking tablet 38850c90b0d94738bfe01d6c60bb473f
I20250110 03:07:38.993865 14414 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 38850c90b0d94738bfe01d6c60bb473f
I20250110 03:07:38.993911 14414 log_verifier.cc:126] Checking tablet 4470d26030b444c9a8ca93562433c0d9
I20250110 03:07:38.999301 14414 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 4470d26030b444c9a8ca93562433c0d9
I20250110 03:07:38.999342 14414 log_verifier.cc:126] Checking tablet 62872e590b72414a81a79e9936027ae8
I20250110 03:07:39.004714 14414 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 62872e590b72414a81a79e9936027ae8
I20250110 03:07:39.004753 14414 log_verifier.cc:126] Checking tablet 78298fadb7eb464caf47cbf44a9ccd61
I20250110 03:07:39.010123 14414 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 78298fadb7eb464caf47cbf44a9ccd61
I20250110 03:07:39.010164 14414 log_verifier.cc:126] Checking tablet 7d92b7bc1b364c8683ed8f52a1445895
I20250110 03:07:39.015664 14414 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 7d92b7bc1b364c8683ed8f52a1445895
I20250110 03:07:39.015705 14414 log_verifier.cc:126] Checking tablet 80982ad268db4d9a9660ddfbda0a1e98
I20250110 03:07:39.021096 14414 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 80982ad268db4d9a9660ddfbda0a1e98
I20250110 03:07:39.021136 14414 log_verifier.cc:126] Checking tablet 8d8d1cd3a6324b4e9cc287d52f646010
I20250110 03:07:39.026497 14414 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 8d8d1cd3a6324b4e9cc287d52f646010
I20250110 03:07:39.026571 14414 log_verifier.cc:126] Checking tablet 9bffd88289a84709b7a93ff6404c9ee8
I20250110 03:07:39.031945 14414 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 9bffd88289a84709b7a93ff6404c9ee8
I20250110 03:07:39.031985 14414 log_verifier.cc:126] Checking tablet a0888f8ca6284065be9f7db277e6e414
I20250110 03:07:39.037340 14414 log_verifier.cc:177] Verified matching terms for 2 ops in tablet a0888f8ca6284065be9f7db277e6e414
I20250110 03:07:39.037382 14414 log_verifier.cc:126] Checking tablet babadeef5e7a40f89435f2803603c9ab
I20250110 03:07:39.042860 14414 log_verifier.cc:177] Verified matching terms for 6 ops in tablet babadeef5e7a40f89435f2803603c9ab
I20250110 03:07:39.042902 14414 log_verifier.cc:126] Checking tablet c5ed9433a60e42d098d28eaa6a016c1e
I20250110 03:07:39.048388 14414 log_verifier.cc:177] Verified matching terms for 5 ops in tablet c5ed9433a60e42d098d28eaa6a016c1e
I20250110 03:07:39.048431 14414 log_verifier.cc:126] Checking tablet fab48f4443a54481b79a36b27d399383
I20250110 03:07:39.053841 14414 log_verifier.cc:177] Verified matching terms for 1 ops in tablet fab48f4443a54481b79a36b27d399383
I20250110 03:07:39.054071 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 32295
I20250110 03:07:39.059444 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 32426
I20250110 03:07:39.064070 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 486
I20250110 03:07:39.068483 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 619
I20250110 03:07:39.072834 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 32204
2025-01-10T03:07:39Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (8557 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (8557 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20250110 03:07:39.103392 14414 test_util.cc:274] Using random seed: 1706494110
I20250110 03:07:39.103847 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:07:39.103876 14414 ts_itest-base.cc:113] --------------
I20250110 03:07:39.103888 14414 ts_itest-base.cc:114] 6 tablet servers
I20250110 03:07:39.103897 14414 ts_itest-base.cc:115] 3 replicas per TS
I20250110 03:07:39.103907 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:07:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:07:39Z Disabled control of system clock
I20250110 03:07:39.110837 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:44425
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:44425
--location_mapping_cmd=/tmp/dist-test-taskWQrDfu/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20250110 03:07:39.183454 1446 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20250110 03:07:39.183617 1446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.183633 1446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.183648 1446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.184808 1446 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.184842 1446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:07:39.184856 1446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.184867 1446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:07:39.184882 1446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:07:39.186156 1446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskWQrDfu/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:44425
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.186313 1446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.186473 1446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.190054 1454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.191044 1452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.192196 1451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.193066 1446 server_base.cc:1034] running on GCE node
I20250110 03:07:39.193249 1446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.193424 1446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.194566 1446 hybrid_clock.cc:648] HybridClock initialized: now 1736478459194530 us; error 32 us; skew 500 ppm
I20250110 03:07:39.195570 1446 webserver.cc:458] Webserver started at http://127.14.19.190:33691/ using document root <none> and password file <none>
I20250110 03:07:39.195726 1446 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.195765 1446 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.195830 1446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.196599 1446 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "38723b9998c1446389654812ab0fb5f2"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.196833 1446 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "38723b9998c1446389654812ab0fb5f2"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.197836 1446 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:39.198398 1460 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.198539 1446 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:39.198637 1446 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "38723b9998c1446389654812ab0fb5f2"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.198693 1446 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:39.213706 1446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:39.213958 1446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:39.214072 1446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:39.217559 1446 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:44425
I20250110 03:07:39.217589 1512 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:44425 every 8 connection(s)
I20250110 03:07:39.217839 1446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:07:39.218339 1513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:39.220422 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2: Bootstrap starting.
I20250110 03:07:39.220911 1513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:39.221117 1513 log.cc:826] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:39.221611 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2: No bootstrap required, opened a new log
I20250110 03:07:39.223022 1513 raft_consensus.cc:357] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38723b9998c1446389654812ab0fb5f2" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 44425 } }
I20250110 03:07:39.223132 1513 raft_consensus.cc:383] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:39.223151 1513 raft_consensus.cc:738] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38723b9998c1446389654812ab0fb5f2, State: Initialized, Role: FOLLOWER
I20250110 03:07:39.223232 1513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38723b9998c1446389654812ab0fb5f2" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 44425 } }
I20250110 03:07:39.223276 1513 raft_consensus.cc:397] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:39.223295 1513 raft_consensus.cc:491] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:39.223325 1513 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:39.223773 1513 raft_consensus.cc:513] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38723b9998c1446389654812ab0fb5f2" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 44425 } }
I20250110 03:07:39.223856 1513 leader_election.cc:304] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38723b9998c1446389654812ab0fb5f2; no voters:
I20250110 03:07:39.223994 1513 leader_election.cc:290] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:39.224042 1516 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:39.224128 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 1446
I20250110 03:07:39.224193 1516 raft_consensus.cc:695] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [term 1 LEADER]: Becoming Leader. State: Replica: 38723b9998c1446389654812ab0fb5f2, State: Running, Role: LEADER
I20250110 03:07:39.224228 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:07:39.224231 1513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:07:39.224327 1516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38723b9998c1446389654812ab0fb5f2" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 44425 } }
I20250110 03:07:39.224768 1518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38723b9998c1446389654812ab0fb5f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38723b9998c1446389654812ab0fb5f2" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 44425 } } }
I20250110 03:07:39.224857 1518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:39.225132 1525 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:07:39.225522 1517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38723b9998c1446389654812ab0fb5f2. Latest consensus state: current_term: 1 leader_uuid: "38723b9998c1446389654812ab0fb5f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38723b9998c1446389654812ab0fb5f2" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 44425 } } }
I20250110 03:07:39.225584 1525 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:07:39.225651 1517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:39.226972 1525 catalog_manager.cc:1348] Generated new cluster ID: 6926936321e84857ab6a5cf57929adb1
I20250110 03:07:39.227030 1525 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:07:39.236461 1525 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:07:39.236879 1525 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:07:39.247658 1525 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 38723b9998c1446389654812ab0fb5f2: Generated new TSK 0
I20250110 03:07:39.247787 1525 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:07:39.252441 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:39.337918 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.338069 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.338097 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.339313 1537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.339354 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.339390 1537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:07:39.340723 1537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.1537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.340942 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.341142 1537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.344918 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.345891 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.347082 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.347909 1537 server_base.cc:1034] running on GCE node
I20250110 03:07:39.348065 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.348261 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.349387 1537 hybrid_clock.cc:648] HybridClock initialized: now 1736478459349369 us; error 37 us; skew 500 ppm
I20250110 03:07:39.350641 1537 webserver.cc:458] Webserver started at http://127.14.19.129:34657/ using document root <none> and password file <none>
I20250110 03:07:39.350821 1537 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.350869 1537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.350968 1537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.351783 1537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.352088 1537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.353229 1537 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:39.353840 1551 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.354009 1537 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:39.354094 1537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.354171 1537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:39.369635 1537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:39.369820 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:39.369920 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:39.370081 1537 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:39.370339 1537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:39.370399 1537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.370435 1537 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:39.370465 1537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.376353 1537 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:33179
I20250110 03:07:39.376394 1664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:33179 every 8 connection(s)
I20250110 03:07:39.376695 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:07:39.377049 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 1537
I20250110 03:07:39.377161 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:07:39.378276 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:39.381769 1665 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:39.381865 1665 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:39.382032 1665 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:39.413519 1477 ts_manager.cc:194] Registered new tserver with Master: baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:39.414216 1477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:48163
W20250110 03:07:39.460316 1668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.460477 1668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.460495 1668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.461735 1668 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.461774 1668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.461795 1668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:07:39.463148 1668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.1668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.463341 1668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.463531 1668 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.467283 1675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.468365 1677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.469147 1674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.470266 1668 server_base.cc:1034] running on GCE node
I20250110 03:07:39.470422 1668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.470623 1668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.471740 1668 hybrid_clock.cc:648] HybridClock initialized: now 1736478459471734 us; error 28 us; skew 500 ppm
I20250110 03:07:39.472774 1668 webserver.cc:458] Webserver started at http://127.14.19.130:44287/ using document root <none> and password file <none>
I20250110 03:07:39.472944 1668 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.473012 1668 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.473111 1668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.473912 1668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "90d44e46d37f48258cea315fd7e589d5"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.474197 1668 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "90d44e46d37f48258cea315fd7e589d5"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.475325 1668 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:07:39.475921 1683 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.476073 1668 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.476140 1668 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "90d44e46d37f48258cea315fd7e589d5"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.476197 1668 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:39.508459 1668 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:39.508683 1668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:39.508787 1668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:39.508970 1668 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:39.509248 1668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:39.509282 1668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.509315 1668 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:39.509343 1668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.514878 1668 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:38231
I20250110 03:07:39.514940 1796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:38231 every 8 connection(s)
I20250110 03:07:39.515239 1668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:07:39.519685 1797 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:39.519764 1797 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:39.519906 1797 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:39.524133 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 1668
I20250110 03:07:39.524235 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:07:39.525955 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:39.553318 1477 ts_manager.cc:194] Registered new tserver with Master: 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231)
I20250110 03:07:39.553939 1477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:53487
W20250110 03:07:39.604501 1801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.604668 1801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.604700 1801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.605858 1801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.605894 1801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.605916 1801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:07:39.607244 1801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.1801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.607395 1801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.607558 1801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.611382 1809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.612391 1807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.613538 1806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.614290 1801 server_base.cc:1034] running on GCE node
I20250110 03:07:39.614418 1801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.614631 1801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.615756 1801 hybrid_clock.cc:648] HybridClock initialized: now 1736478459615737 us; error 36 us; skew 500 ppm
I20250110 03:07:39.616832 1801 webserver.cc:458] Webserver started at http://127.14.19.131:38345/ using document root <none> and password file <none>
I20250110 03:07:39.617015 1801 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.617062 1801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.617157 1801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.617950 1801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.618225 1801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.619347 1801 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250110 03:07:39.619920 1815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.620067 1801 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:39.620127 1801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.620198 1801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:39.631405 1801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:39.631565 1801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:39.631654 1801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:39.631778 1801 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:39.632072 1801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:39.632103 1801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.632136 1801 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:39.632165 1801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.637207 1801 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:42095
I20250110 03:07:39.637265 1928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:42095 every 8 connection(s)
I20250110 03:07:39.637495 1801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:07:39.641489 1929 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:39.641566 1929 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:39.641733 1929 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:39.643142 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 1801
I20250110 03:07:39.643236 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:07:39.644812 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:39.685149 1477 ts_manager.cc:194] Registered new tserver with Master: 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:39.685657 1477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:33251
W20250110 03:07:39.735124 1933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.735265 1933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.735284 1933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.736547 1933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.736584 1933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.736605 1933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:39.738090 1933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.1933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.738288 1933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.738487 1933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.742413 1941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.743330 1938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.744359 1939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.745199 1933 server_base.cc:1034] running on GCE node
I20250110 03:07:39.745352 1933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.745545 1933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.746675 1933 hybrid_clock.cc:648] HybridClock initialized: now 1736478459746654 us; error 33 us; skew 500 ppm
I20250110 03:07:39.747908 1933 webserver.cc:458] Webserver started at http://127.14.19.132:45285/ using document root <none> and password file <none>
I20250110 03:07:39.748128 1933 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.748178 1933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.748268 1933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.749089 1933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "856267e243cb46ab884b7a1694ae465e"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.749372 1933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "856267e243cb46ab884b7a1694ae465e"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.750464 1933 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20250110 03:07:39.751066 1947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.751223 1933 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.751294 1933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "856267e243cb46ab884b7a1694ae465e"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.751353 1933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:39.765458 1933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:39.765643 1933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:39.765745 1933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:39.765920 1933 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:39.766188 1933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:39.766218 1933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.766240 1933 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:39.766269 1933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.771607 1933 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:40673
I20250110 03:07:39.771672 2060 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:40673 every 8 connection(s)
I20250110 03:07:39.771914 1933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:39.773423 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 1933
I20250110 03:07:39.773514 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:07:39.774641 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:0
--local_ip_for_outbound_sockets=127.14.19.133
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:39.776597 2061 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:39.776679 2061 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:39.776844 2061 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:39.816547 1477 ts_manager.cc:194] Registered new tserver with Master: 856267e243cb46ab884b7a1694ae465e (127.14.19.132:40673)
I20250110 03:07:39.817028 1477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:55377
W20250110 03:07:39.850834 2064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.850980 2064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.851009 2064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.852243 2064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.852290 2064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.852326 2064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:07:39.853700 2064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.2064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.853955 2064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.854180 2064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.858978 2070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.859154 2073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.860261 2071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.861099 2064 server_base.cc:1034] running on GCE node
I20250110 03:07:39.861263 2064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.861447 2064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.862586 2064 hybrid_clock.cc:648] HybridClock initialized: now 1736478459862570 us; error 30 us; skew 500 ppm
I20250110 03:07:39.863631 2064 webserver.cc:458] Webserver started at http://127.14.19.133:41981/ using document root <none> and password file <none>
I20250110 03:07:39.863804 2064 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.863848 2064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.863942 2064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.864763 2064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c2867695ca20469dba7d61ebc5ecebfa"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.865051 2064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c2867695ca20469dba7d61ebc5ecebfa"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.866113 2064 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:39.866695 2079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.866844 2064 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.866912 2064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c2867695ca20469dba7d61ebc5ecebfa"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.866971 2064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:39.884953 2064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:39.885152 2064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:39.885255 2064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:39.885416 2064 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:39.885684 2064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:39.885715 2064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.885746 2064 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:39.885771 2064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.891047 2064 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:36633
I20250110 03:07:39.891104 2192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:36633 every 8 connection(s)
I20250110 03:07:39.891359 2064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:07:39.896013 2193 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:39.896090 2193 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:39.896235 2193 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:39.899176 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 2064
I20250110 03:07:39.899263 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 03:07:39.900429 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:0
--local_ip_for_outbound_sockets=127.14.19.134
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:39.928025 1477 ts_manager.cc:194] Registered new tserver with Master: c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633)
I20250110 03:07:39.928534 1477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:55905
W20250110 03:07:39.975234 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:39.975370 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:39.975389 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:39.976584 2197 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:39.976641 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:39.976665 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:07:39.977970 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=0
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:39.978173 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:39.978360 2197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:39.982239 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.983294 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:39.984088 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:39.985148 2197 server_base.cc:1034] running on GCE node
I20250110 03:07:39.985292 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:39.985481 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:39.986600 2197 hybrid_clock.cc:648] HybridClock initialized: now 1736478459986588 us; error 26 us; skew 500 ppm
I20250110 03:07:39.987821 2197 webserver.cc:458] Webserver started at http://127.14.19.134:44523/ using document root <none> and password file <none>
I20250110 03:07:39.988018 2197 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:39.988076 2197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:39.988164 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:39.988979 2197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.989259 2197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.990365 2197 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:39.991016 2211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.991170 2197 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:39.991240 2197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:39.991295 2197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:40.003578 2197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:40.003772 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:40.003875 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:40.004032 2197 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:40.004323 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:40.004356 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:40.004386 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:40.004408 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:40.009538 2197 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:38075
I20250110 03:07:40.009600 2324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:38075 every 8 connection(s)
I20250110 03:07:40.011396 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:07:40.015261 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 2197
I20250110 03:07:40.015364 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20250110 03:07:40.018220 2325 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:40.018322 2325 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:40.018491 2325 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:40.050940 1477 ts_manager.cc:194] Registered new tserver with Master: 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075)
I20250110 03:07:40.051414 1477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:39739
I20250110 03:07:40.059958 14414 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250110 03:07:40.065673 14414 external_mini_cluster.cc:934] 6 TS(s) registered with all masters
I20250110 03:07:40.065725 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 2197
I20250110 03:07:40.070875 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 2064
I20250110 03:07:40.075096 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 1933
I20250110 03:07:40.415217 1665 heartbeater.cc:502] Master 127.14.19.190:44425 was elected leader, sending a full tablet report...
I20250110 03:07:40.555002 1797 heartbeater.cc:502] Master 127.14.19.190:44425 was elected leader, sending a full tablet report...
I20250110 03:07:40.686623 1929 heartbeater.cc:502] Master 127.14.19.190:44425 was elected leader, sending a full tablet report...
I20250110 03:07:43.829852 1477 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48416:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250110 03:07:43.831877 1477 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 03:07:43.838392 1730 tablet_service.cc:1467] Processing CreateTablet for tablet 6fde79de661a4993ace77eadfc877bf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.838451 1861 tablet_service.cc:1467] Processing CreateTablet for tablet b373739a50754ecb8753a84d813c0151 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.838456 1863 tablet_service.cc:1467] Processing CreateTablet for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.838773 1730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fde79de661a4993ace77eadfc877bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.838771 1862 tablet_service.cc:1467] Processing CreateTablet for tablet 6fde79de661a4993ace77eadfc877bf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.838984 1599 tablet_service.cc:1467] Processing CreateTablet for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.839123 1597 tablet_service.cc:1467] Processing CreateTablet for tablet b373739a50754ecb8753a84d813c0151 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.839303 1599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.839452 1597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b373739a50754ecb8753a84d813c0151. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.841650 2344 tablet_bootstrap.cc:492] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:43.838389 1731 tablet_service.cc:1467] Processing CreateTablet for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.841868 1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.839015 1598 tablet_service.cc:1467] Processing CreateTablet for tablet 6fde79de661a4993ace77eadfc877bf4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.842146 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fde79de661a4993ace77eadfc877bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.842571 2344 tablet_bootstrap.cc:654] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.842677 1729 tablet_service.cc:1467] Processing CreateTablet for tablet b373739a50754ecb8753a84d813c0151 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:43.842783 1729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b373739a50754ecb8753a84d813c0151. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.842885 2344 log.cc:826] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:43.843676 2344 tablet_bootstrap.cc:492] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:43.843742 2344 ts_tablet_manager.cc:1397] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:43.838773 1861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b373739a50754ecb8753a84d813c0151. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.844389 1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.845450 2345 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:43.845535 2344 raft_consensus.cc:357] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.845676 2344 raft_consensus.cc:383] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.845717 2344 raft_consensus.cc:738] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.845829 2344 consensus_queue.cc:260] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.845993 2345 tablet_bootstrap.cc:654] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.846087 2344 ts_tablet_manager.cc:1428] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250110 03:07:43.846227 2344 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:43.846271 2345 log.cc:826] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:43.846603 1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fde79de661a4993ace77eadfc877bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:43.846688 2344 tablet_bootstrap.cc:654] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.847251 2345 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:43.847294 2344 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:43.847308 2345 ts_tablet_manager.cc:1397] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:43.847338 2344 ts_tablet_manager.cc:1397] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:43.847443 2350 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:43.847496 2344 raft_consensus.cc:357] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:43.847553 2344 raft_consensus.cc:383] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.847574 2344 raft_consensus.cc:738] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.847645 2344 consensus_queue.cc:260] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:43.847810 2344 ts_tablet_manager.cc:1428] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:43.847868 2344 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:43.848029 2350 tablet_bootstrap.cc:654] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.848259 2344 tablet_bootstrap.cc:654] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.848346 2350 log.cc:826] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:43.848910 2344 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:43.848944 2344 ts_tablet_manager.cc:1397] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:43.849040 2344 raft_consensus.cc:357] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.849077 2344 raft_consensus.cc:383] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.849092 2344 raft_consensus.cc:738] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.849123 2344 consensus_queue.cc:260] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.849215 2344 ts_tablet_manager.cc:1428] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:43.849622 2345 raft_consensus.cc:357] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.849758 2345 raft_consensus.cc:383] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.849799 2345 raft_consensus.cc:738] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.849884 2345 consensus_queue.cc:260] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.850131 2345 ts_tablet_manager.cc:1428] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:07:43.850201 2345 tablet_bootstrap.cc:492] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:43.850260 2350 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:43.850301 2350 ts_tablet_manager.cc:1397] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:43.850617 2345 tablet_bootstrap.cc:654] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.851372 2345 tablet_bootstrap.cc:492] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:43.851414 2345 ts_tablet_manager.cc:1397] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:43.851532 2345 raft_consensus.cc:357] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.851577 2345 raft_consensus.cc:383] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.851600 2345 raft_consensus.cc:738] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.851655 2345 consensus_queue.cc:260] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.851743 2345 ts_tablet_manager.cc:1428] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:43.851809 2345 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:43.852010 2350 raft_consensus.cc:357] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:43.852144 2350 raft_consensus.cc:383] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.852191 2345 tablet_bootstrap.cc:654] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.852200 2350 raft_consensus.cc:738] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.852293 2350 consensus_queue.cc:260] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:43.852545 2350 ts_tablet_manager.cc:1428] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 03:07:43.852638 2350 tablet_bootstrap.cc:492] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:43.852831 2345 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:43.852880 2345 ts_tablet_manager.cc:1397] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:43.853030 2345 raft_consensus.cc:357] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:43.853086 2345 raft_consensus.cc:383] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.853104 2345 raft_consensus.cc:738] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.853106 2350 tablet_bootstrap.cc:654] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.853163 2345 consensus_queue.cc:260] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:43.853273 2345 ts_tablet_manager.cc:1428] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:43.853685 2350 tablet_bootstrap.cc:492] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:43.853719 2350 ts_tablet_manager.cc:1397] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:43.853842 2350 raft_consensus.cc:357] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.853901 2350 raft_consensus.cc:383] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.853914 2350 raft_consensus.cc:738] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.853950 2350 consensus_queue.cc:260] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.854032 2350 ts_tablet_manager.cc:1428] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:43.854127 2350 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:43.854431 2350 tablet_bootstrap.cc:654] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:43.854969 2350 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:43.855006 2350 ts_tablet_manager.cc:1397] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:43.855113 2350 raft_consensus.cc:357] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.855155 2350 raft_consensus.cc:383] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:43.855178 2350 raft_consensus.cc:738] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:43.855243 2350 consensus_queue.cc:260] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.855345 2350 ts_tablet_manager.cc:1428] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20250110 03:07:43.880270 1666 tablet.cc:2367] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:07:43.890405 1930 tablet.cc:2367] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:43.986217 2351 raft_consensus.cc:491] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:43.986335 2351 raft_consensus.cc:513] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.986626 2351 leader_election.cc:290] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:43.989310 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b373739a50754ecb8753a84d813c0151" candidate_uuid: "90d44e46d37f48258cea315fd7e589d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:43.989320 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b373739a50754ecb8753a84d813c0151" candidate_uuid: "90d44e46d37f48258cea315fd7e589d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" is_pre_election: true
I20250110 03:07:43.989435 1883 raft_consensus.cc:2463] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d44e46d37f48258cea315fd7e589d5 in term 0.
I20250110 03:07:43.989449 1619 raft_consensus.cc:2463] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d44e46d37f48258cea315fd7e589d5 in term 0.
I20250110 03:07:43.989639 1684 leader_election.cc:304] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, 90d44e46d37f48258cea315fd7e589d5; no voters:
I20250110 03:07:43.989778 2351 raft_consensus.cc:2798] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:43.989851 2351 raft_consensus.cc:491] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:43.989868 2351 raft_consensus.cc:3054] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:43.990497 2351 raft_consensus.cc:513] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.990624 2351 leader_election.cc:290] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [CANDIDATE]: Term 1 election: Requested vote from peers baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:43.990829 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b373739a50754ecb8753a84d813c0151" candidate_uuid: "90d44e46d37f48258cea315fd7e589d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
I20250110 03:07:43.990824 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b373739a50754ecb8753a84d813c0151" candidate_uuid: "90d44e46d37f48258cea315fd7e589d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:43.990903 1619 raft_consensus.cc:3054] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:43.990904 1883 raft_consensus.cc:3054] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:43.991683 1883 raft_consensus.cc:2463] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d44e46d37f48258cea315fd7e589d5 in term 1.
I20250110 03:07:43.991684 1619 raft_consensus.cc:2463] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d44e46d37f48258cea315fd7e589d5 in term 1.
I20250110 03:07:43.991858 1684 leader_election.cc:304] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, 90d44e46d37f48258cea315fd7e589d5; no voters:
I20250110 03:07:43.991972 2351 raft_consensus.cc:2798] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:43.992036 2351 raft_consensus.cc:695] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEADER]: Becoming Leader. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Running, Role: LEADER
I20250110 03:07:43.992118 2351 consensus_queue.cc:237] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:43.992822 1477 catalog_manager.cc:5526] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "90d44e46d37f48258cea315fd7e589d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:44.018155 1798 tablet.cc:2367] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:44.079214 2354 raft_consensus.cc:491] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.079308 2354 raft_consensus.cc:513] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:44.079596 2354 leader_election.cc:290] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095), 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231)
I20250110 03:07:44.082280 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02da6c3e2d4db5b1db2f1de6824bc5" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:44.082389 1883 raft_consensus.cc:2463] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.082403 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02da6c3e2d4db5b1db2f1de6824bc5" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.082513 1751 raft_consensus.cc:2463] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.082603 1552 leader_election.cc:304] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.082752 2354 raft_consensus.cc:2798] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.082818 2354 raft_consensus.cc:491] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.082834 2354 raft_consensus.cc:3054] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.083441 2354 raft_consensus.cc:513] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:44.083559 2354 leader_election.cc:290] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Requested vote from peers 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095), 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231)
I20250110 03:07:44.083709 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02da6c3e2d4db5b1db2f1de6824bc5" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.083706 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02da6c3e2d4db5b1db2f1de6824bc5" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:44.083779 1883 raft_consensus.cc:3054] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.083779 1751 raft_consensus.cc:3054] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.084453 1751 raft_consensus.cc:2463] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.084453 1883 raft_consensus.cc:2463] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.084612 1552 leader_election.cc:304] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.084721 2354 raft_consensus.cc:2798] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.084877 2354 raft_consensus.cc:695] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Leader. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:44.084966 2354 consensus_queue.cc:237] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } }
I20250110 03:07:44.085811 1477 catalog_manager.cc:5526] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:44.206009 2356 raft_consensus.cc:491] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.206104 2356 raft_consensus.cc:513] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.206399 2356 leader_election.cc:290] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179), 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231)
I20250110 03:07:44.209017 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde79de661a4993ace77eadfc877bf4" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" is_pre_election: true
I20250110 03:07:44.209031 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde79de661a4993ace77eadfc877bf4" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.209103 1751 raft_consensus.cc:2463] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 0.
I20250110 03:07:44.209102 1619 raft_consensus.cc:2463] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 0.
I20250110 03:07:44.209265 1817 leader_election.cc:304] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, 90d44e46d37f48258cea315fd7e589d5; no voters:
I20250110 03:07:44.209426 2356 raft_consensus.cc:2798] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.209471 2356 raft_consensus.cc:491] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.209492 2356 raft_consensus.cc:3054] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.210116 2356 raft_consensus.cc:513] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.210242 2356 leader_election.cc:290] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 election: Requested vote from peers baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179), 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231)
I20250110 03:07:44.210400 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde79de661a4993ace77eadfc877bf4" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
I20250110 03:07:44.210397 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fde79de661a4993ace77eadfc877bf4" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.210466 1619 raft_consensus.cc:3054] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.210466 1751 raft_consensus.cc:3054] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.211169 1619 raft_consensus.cc:2463] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 1.
I20250110 03:07:44.211169 1751 raft_consensus.cc:2463] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 1.
I20250110 03:07:44.211352 1818 leader_election.cc:304] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.211470 2356 raft_consensus.cc:2798] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.211587 2356 raft_consensus.cc:695] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Becoming Leader. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Running, Role: LEADER
I20250110 03:07:44.211665 2356 consensus_queue.cc:237] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.212360 1477 catalog_manager.cc:5526] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } }
W20250110 03:07:44.349054 2371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:44.349201 2371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:44.350407 2371 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250110 03:07:44.384899 1619 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.385068 1883 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.385140 2351 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.385295 2351 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.385545 1751 raft_consensus.cc:1270] T 6fde79de661a4993ace77eadfc877bf4 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.386341 1882 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.386695 2365 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.386816 1751 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.386945 1619 raft_consensus.cc:1270] T 6fde79de661a4993ace77eadfc877bf4 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.387239 2369 consensus_queue.cc:1035] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.387503 2356 consensus_queue.cc:1035] T 6fde79de661a4993ace77eadfc877bf4 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.387929 2365 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.390156 2408 mvcc.cc:204] Tried to move back new op lower bound from 7112615790115901440 to 7112615788512153600. Current Snapshot: MvccSnapshot[applied={T|T < 7112615790115901440}]
I20250110 03:07:44.390266 2406 mvcc.cc:204] Tried to move back new op lower bound from 7112615790126501888 to 7112615789411475456. Current Snapshot: MvccSnapshot[applied={T|T < 7112615790119653376}]
I20250110 03:07:44.392828 2404 mvcc.cc:204] Tried to move back new op lower bound from 7112615790145347584 to 7112615788512153600. Current Snapshot: MvccSnapshot[applied={T|T < 7112615790115901440}]
INSERT report
rows total: 2000
time total: 25.1071 ms
time per row: 0.0125535 ms
I20250110 03:07:44.403000 1477 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48416:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250110 03:07:44.403127 1477 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 03:07:44.404438 1599 tablet_service.cc:1467] Processing CreateTablet for tablet d68741aa20be48229a19dda143743bca (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.404460 1729 tablet_service.cc:1467] Processing CreateTablet for tablet d68741aa20be48229a19dda143743bca (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.404469 1730 tablet_service.cc:1467] Processing CreateTablet for tablet 48cdbe84b6094cccaf69b553fd8f1d68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.404551 1729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d68741aa20be48229a19dda143743bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.404551 1599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d68741aa20be48229a19dda143743bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.404597 1730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cdbe84b6094cccaf69b553fd8f1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.404743 1598 tablet_service.cc:1467] Processing CreateTablet for tablet 48cdbe84b6094cccaf69b553fd8f1d68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.404753 1731 tablet_service.cc:1467] Processing CreateTablet for tablet 56aea05a8a714b4ca357e24049c7f348 (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.404825 1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aea05a8a714b4ca357e24049c7f348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.404826 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cdbe84b6094cccaf69b553fd8f1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.405915 1597 tablet_service.cc:1467] Processing CreateTablet for tablet 56aea05a8a714b4ca357e24049c7f348 (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.406004 1597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aea05a8a714b4ca357e24049c7f348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.406818 2426 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:44.406836 1862 tablet_service.cc:1467] Processing CreateTablet for tablet 48cdbe84b6094cccaf69b553fd8f1d68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.406862 1861 tablet_service.cc:1467] Processing CreateTablet for tablet d68741aa20be48229a19dda143743bca (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.406941 1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cdbe84b6094cccaf69b553fd8f1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.407032 1863 tablet_service.cc:1467] Processing CreateTablet for tablet 56aea05a8a714b4ca357e24049c7f348 (DEFAULT_TABLE table=rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.407059 2427 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:44.407130 1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aea05a8a714b4ca357e24049c7f348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.407449 2426 tablet_bootstrap.cc:654] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.407606 2427 tablet_bootstrap.cc:654] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.408329 1861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d68741aa20be48229a19dda143743bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.408452 2430 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:44.408663 2427 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:44.408737 2427 ts_tablet_manager.cc:1397] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:44.408859 2430 tablet_bootstrap.cc:654] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.408995 2427 raft_consensus.cc:357] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.409128 2427 raft_consensus.cc:383] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.409164 2427 raft_consensus.cc:738] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.409221 2427 consensus_queue.cc:260] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.409335 2427 ts_tablet_manager.cc:1428] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:44.409382 2427 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:44.409397 2426 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:44.409437 2426 ts_tablet_manager.cc:1397] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:44.409551 2426 raft_consensus.cc:357] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.409591 2426 raft_consensus.cc:383] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.409603 2426 raft_consensus.cc:738] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.409637 2426 consensus_queue.cc:260] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.409729 2426 ts_tablet_manager.cc:1428] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.409736 2427 tablet_bootstrap.cc:654] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.409775 2426 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:44.410077 2426 tablet_bootstrap.cc:654] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.410271 2427 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:44.410323 2427 ts_tablet_manager.cc:1397] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:44.410449 2427 raft_consensus.cc:357] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.410518 2427 raft_consensus.cc:383] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.410584 2427 raft_consensus.cc:738] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.410632 2427 consensus_queue.cc:260] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.410706 2427 ts_tablet_manager.cc:1428] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.410768 2427 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:44.411010 2426 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:44.411053 2426 ts_tablet_manager.cc:1397] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:44.411223 2427 tablet_bootstrap.cc:654] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.411222 2426 raft_consensus.cc:357] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.411278 2426 raft_consensus.cc:383] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.411303 2426 raft_consensus.cc:738] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.411348 2426 consensus_queue.cc:260] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.411454 2426 ts_tablet_manager.cc:1428] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.411516 2426 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:44.411664 2430 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:44.411705 2430 ts_tablet_manager.cc:1397] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:44.411809 2427 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:44.411859 2427 ts_tablet_manager.cc:1397] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.411914 2426 tablet_bootstrap.cc:654] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.411875 2430 raft_consensus.cc:357] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.411960 2430 raft_consensus.cc:383] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.411985 2430 raft_consensus.cc:738] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.412034 2430 consensus_queue.cc:260] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.412025 2427 raft_consensus.cc:357] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.412081 2427 raft_consensus.cc:383] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.412101 2427 raft_consensus.cc:738] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.412143 2427 consensus_queue.cc:260] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.412220 2430 ts_tablet_manager.cc:1428] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:44.412245 2427 ts_tablet_manager.cc:1428] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:44.412286 2430 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:44.412635 2426 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:44.412686 2426 ts_tablet_manager.cc:1397] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:44.412942 2426 raft_consensus.cc:357] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.413030 2426 raft_consensus.cc:383] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.413059 2426 raft_consensus.cc:738] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.413060 2430 tablet_bootstrap.cc:654] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.413103 2426 consensus_queue.cc:260] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.413223 2426 ts_tablet_manager.cc:1428] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:44.413663 2430 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:44.413715 2430 ts_tablet_manager.cc:1397] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:44.413842 2430 raft_consensus.cc:357] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.413904 2430 raft_consensus.cc:383] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.413925 2430 raft_consensus.cc:738] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.413980 2430 consensus_queue.cc:260] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.414059 2430 ts_tablet_manager.cc:1428] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.414114 2430 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:44.414522 2430 tablet_bootstrap.cc:654] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.415024 2430 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:44.415058 2430 ts_tablet_manager.cc:1397] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.415158 2430 raft_consensus.cc:357] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.415205 2430 raft_consensus.cc:383] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.415217 2430 raft_consensus.cc:738] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.415248 2430 consensus_queue.cc:260] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.415350 2430 ts_tablet_manager.cc:1428] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.556923 2365 raft_consensus.cc:491] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.557016 2365 raft_consensus.cc:513] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.557155 2365 leader_election.cc:290] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.557359 1882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56aea05a8a714b4ca357e24049c7f348" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:44.557358 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56aea05a8a714b4ca357e24049c7f348" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.557452 1751 raft_consensus.cc:2463] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.557452 1882 raft_consensus.cc:2463] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.557627 1553 leader_election.cc:304] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.557716 2365 raft_consensus.cc:2798] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.557763 2365 raft_consensus.cc:491] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.557777 2365 raft_consensus.cc:3054] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.558393 2365 raft_consensus.cc:513] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.558493 2365 leader_election.cc:290] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.558670 1882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56aea05a8a714b4ca357e24049c7f348" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:44.558683 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56aea05a8a714b4ca357e24049c7f348" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.558746 1751 raft_consensus.cc:3054] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.558746 1882 raft_consensus.cc:3054] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.559410 1882 raft_consensus.cc:2463] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.559410 1751 raft_consensus.cc:2463] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.559580 1552 leader_election.cc:304] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.559666 2365 raft_consensus.cc:2798] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.559725 2365 raft_consensus.cc:695] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Leader. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:44.559778 2365 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.560276 1477 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:44.689489 2365 raft_consensus.cc:491] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.689566 2365 raft_consensus.cc:513] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.689703 2365 leader_election.cc:290] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.689890 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68741aa20be48229a19dda143743bca" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.689899 1882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68741aa20be48229a19dda143743bca" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:44.689967 1751 raft_consensus.cc:2463] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.689990 1882 raft_consensus.cc:2463] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.690100 1553 leader_election.cc:304] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.690200 2365 raft_consensus.cc:2798] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.690255 2365 raft_consensus.cc:491] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.690279 2365 raft_consensus.cc:3054] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.690922 2365 raft_consensus.cc:513] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.691049 2365 leader_election.cc:290] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.691205 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68741aa20be48229a19dda143743bca" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.691236 1882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68741aa20be48229a19dda143743bca" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:44.691277 1751 raft_consensus.cc:3054] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.691303 1882 raft_consensus.cc:3054] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.691792 1751 raft_consensus.cc:2463] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.691866 1882 raft_consensus.cc:2463] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.691972 1553 leader_election.cc:304] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.692055 2365 raft_consensus.cc:2798] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.692111 2365 raft_consensus.cc:695] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Leader. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:44.692167 2365 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.692723 1477 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:44.693178 2410 raft_consensus.cc:491] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.693238 2410 raft_consensus.cc:513] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.693372 2410 leader_election.cc:290] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:44.693547 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cdbe84b6094cccaf69b553fd8f1d68" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" is_pre_election: true
I20250110 03:07:44.693614 1619 raft_consensus.cc:2463] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 0.
I20250110 03:07:44.693683 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cdbe84b6094cccaf69b553fd8f1d68" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.693744 1751 raft_consensus.cc:2463] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 0.
I20250110 03:07:44.693735 1818 leader_election.cc:304] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.693812 2410 raft_consensus.cc:2798] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.693847 2410 raft_consensus.cc:491] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.693866 2410 raft_consensus.cc:3054] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.694358 2410 raft_consensus.cc:513] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.694459 2410 leader_election.cc:290] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:44.694612 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cdbe84b6094cccaf69b553fd8f1d68" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.694630 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48cdbe84b6094cccaf69b553fd8f1d68" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
I20250110 03:07:44.694669 1751 raft_consensus.cc:3054] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.694689 1619 raft_consensus.cc:3054] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.695122 1751 raft_consensus.cc:2463] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 1.
I20250110 03:07:44.695134 1619 raft_consensus.cc:2463] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 1.
I20250110 03:07:44.695272 1818 leader_election.cc:304] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.695358 2410 raft_consensus.cc:2798] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.695403 2410 raft_consensus.cc:695] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Becoming Leader. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Running, Role: LEADER
I20250110 03:07:44.695430 2410 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.696101 1477 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } }
W20250110 03:07:44.831861 2441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:44.832017 2441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:44.833189 2441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250110 03:07:44.862798 1751 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:44.862943 1750 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.862949 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.863113 1619 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:44.863142 2365 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.863292 1750 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.863348 2356 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.863533 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:44.863533 2468 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.863770 2354 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.863960 2467 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:44.864053 2365 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 21.2635 ms
time per row: 0.0106317 ms
I20250110 03:07:44.879153 1470 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48416:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250110 03:07:44.879313 1470 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 03:07:44.880553 1597 tablet_service.cc:1467] Processing CreateTablet for tablet eefff01740a34c40ba687c7242b87770 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.880591 1731 tablet_service.cc:1467] Processing CreateTablet for tablet a4054e1b6e654bc4b57d42be2a85296e (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.880591 1729 tablet_service.cc:1467] Processing CreateTablet for tablet eefff01740a34c40ba687c7242b87770 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.880681 1597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefff01740a34c40ba687c7242b87770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.880617 1730 tablet_service.cc:1467] Processing CreateTablet for tablet d0eee4c5c9b54dfca9b59276787773db (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.880731 1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4054e1b6e654bc4b57d42be2a85296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.880846 1729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefff01740a34c40ba687c7242b87770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.881004 1730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0eee4c5c9b54dfca9b59276787773db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.882007 2426 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:44.881989 1599 tablet_service.cc:1467] Processing CreateTablet for tablet d0eee4c5c9b54dfca9b59276787773db (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.882006 1598 tablet_service.cc:1467] Processing CreateTablet for tablet a4054e1b6e654bc4b57d42be2a85296e (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.882077 1599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0eee4c5c9b54dfca9b59276787773db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.882227 2427 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:44.882398 2426 tablet_bootstrap.cc:654] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.882447 2427 tablet_bootstrap.cc:654] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.882680 1598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4054e1b6e654bc4b57d42be2a85296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.882675 1861 tablet_service.cc:1467] Processing CreateTablet for tablet eefff01740a34c40ba687c7242b87770 (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.882711 1862 tablet_service.cc:1467] Processing CreateTablet for tablet d0eee4c5c9b54dfca9b59276787773db (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.882798 1861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefff01740a34c40ba687c7242b87770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.882894 1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0eee4c5c9b54dfca9b59276787773db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.883373 2426 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:44.883430 2426 ts_tablet_manager.cc:1397] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.883549 2426 raft_consensus.cc:357] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.883603 2426 raft_consensus.cc:383] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.883621 2426 raft_consensus.cc:738] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.883656 2426 consensus_queue.cc:260] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.883730 2426 ts_tablet_manager.cc:1428] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.883774 2426 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:44.884068 1863 tablet_service.cc:1467] Processing CreateTablet for tablet a4054e1b6e654bc4b57d42be2a85296e (DEFAULT_TABLE table=rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:44.884112 2426 tablet_bootstrap.cc:654] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.884147 1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4054e1b6e654bc4b57d42be2a85296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:44.884263 2427 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:44.884294 2427 ts_tablet_manager.cc:1397] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20250110 03:07:44.884402 2427 raft_consensus.cc:357] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.884469 2427 raft_consensus.cc:383] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.884492 2427 raft_consensus.cc:738] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.884534 2427 consensus_queue.cc:260] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.884594 2426 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:44.884630 2426 ts_tablet_manager.cc:1397] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.884629 2427 ts_tablet_manager.cc:1428] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.884690 2427 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:44.884758 2426 raft_consensus.cc:357] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.884827 2426 raft_consensus.cc:383] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.884850 2426 raft_consensus.cc:738] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.884894 2426 consensus_queue.cc:260] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.884995 2426 ts_tablet_manager.cc:1428] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.885062 2426 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:44.885079 2427 tablet_bootstrap.cc:654] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.885218 2430 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:44.885563 2426 tablet_bootstrap.cc:654] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.885634 2430 tablet_bootstrap.cc:654] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.886059 2427 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:44.886090 2426 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: No bootstrap required, opened a new log
I20250110 03:07:44.886111 2427 ts_tablet_manager.cc:1397] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:44.886121 2426 ts_tablet_manager.cc:1397] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.886268 2426 raft_consensus.cc:357] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.886262 2427 raft_consensus.cc:357] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.886323 2426 raft_consensus.cc:383] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.886343 2427 raft_consensus.cc:383] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.886371 2426 raft_consensus.cc:738] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.886371 2427 raft_consensus.cc:738] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.886412 2427 consensus_queue.cc:260] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.886411 2426 consensus_queue.cc:260] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.886490 2426 ts_tablet_manager.cc:1428] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.886493 2427 ts_tablet_manager.cc:1428] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.887274 2427 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:44.887476 2430 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:44.887519 2430 ts_tablet_manager.cc:1397] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:44.887668 2427 tablet_bootstrap.cc:654] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.887656 2430 raft_consensus.cc:357] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.887697 2430 raft_consensus.cc:383] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.887709 2430 raft_consensus.cc:738] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.887743 2430 consensus_queue.cc:260] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.887849 2430 ts_tablet_manager.cc:1428] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.887895 2430 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:44.888273 2430 tablet_bootstrap.cc:654] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.888437 2427 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: No bootstrap required, opened a new log
I20250110 03:07:44.888482 2427 ts_tablet_manager.cc:1397] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:44.888599 2427 raft_consensus.cc:357] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.888664 2427 raft_consensus.cc:383] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.888691 2427 raft_consensus.cc:738] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.888743 2427 consensus_queue.cc:260] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.888833 2427 ts_tablet_manager.cc:1428] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.888936 2430 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:44.888989 2430 ts_tablet_manager.cc:1397] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.889117 2430 raft_consensus.cc:357] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.889184 2430 raft_consensus.cc:383] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.889209 2430 raft_consensus.cc:738] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.889258 2430 consensus_queue.cc:260] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.889353 2430 ts_tablet_manager.cc:1428] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.889420 2430 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: Bootstrap starting.
I20250110 03:07:44.889891 2430 tablet_bootstrap.cc:654] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:44.890472 2430 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: No bootstrap required, opened a new log
I20250110 03:07:44.890513 2430 ts_tablet_manager.cc:1397] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:44.890681 2430 raft_consensus.cc:357] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.890731 2430 raft_consensus.cc:383] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:44.890754 2430 raft_consensus.cc:738] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Initialized, Role: FOLLOWER
I20250110 03:07:44.890823 2430 consensus_queue.cc:260] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.890904 2430 ts_tablet_manager.cc:1428] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:44.940655 2468 raft_consensus.cc:491] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.940732 2468 raft_consensus.cc:513] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.940869 2468 leader_election.cc:290] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.941030 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.941064 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:44.941112 1751 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.941125 1883 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.941269 1552 leader_election.cc:304] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.941377 2468 raft_consensus.cc:2798] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.941427 2468 raft_consensus.cc:491] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.941453 2468 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.942057 2468 raft_consensus.cc:513] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.942176 2468 leader_election.cc:290] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.942318 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.942322 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:44.942374 1883 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.942373 1751 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.942950 1751 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.943017 1883 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.943202 1553 leader_election.cc:304] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.943332 2468 raft_consensus.cc:2798] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.943393 2468 raft_consensus.cc:695] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Leader. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:44.943449 2468 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.943938 1468 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:44.960947 2468 raft_consensus.cc:491] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:44.960999 2468 raft_consensus.cc:513] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.961090 2468 leader_election.cc:290] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.961262 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:44.961280 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:44.961326 1883 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.961351 1751 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:44.961495 1552 leader_election.cc:304] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.961583 2468 raft_consensus.cc:2798] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:44.961627 2468 raft_consensus.cc:491] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:44.961642 2468 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.962112 2468 raft_consensus.cc:513] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.962218 2468 leader_election.cc:290] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:44.962349 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:44.962370 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:44.962409 1751 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.962424 1883 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:44.962905 1751 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.962968 1883 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:44.963063 1553 leader_election.cc:304] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:44.963142 2468 raft_consensus.cc:2798] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:44.963191 2468 raft_consensus.cc:695] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Leader. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:44.963232 2468 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:44.963680 1468 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:45.011605 2468 raft_consensus.cc:491] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:45.011675 2468 raft_consensus.cc:513] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:45.011801 2468 leader_election.cc:290] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:45.011977 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4054e1b6e654bc4b57d42be2a85296e" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5" is_pre_election: true
I20250110 03:07:45.012043 1751 raft_consensus.cc:2463] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:45.012017 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4054e1b6e654bc4b57d42be2a85296e" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" is_pre_election: true
I20250110 03:07:45.012109 1883 raft_consensus.cc:2463] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 0.
I20250110 03:07:45.012229 1553 leader_election.cc:304] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:45.012323 2468 raft_consensus.cc:2798] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:45.012351 2468 raft_consensus.cc:491] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:45.012364 2468 raft_consensus.cc:3054] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:45.014084 2468 raft_consensus.cc:513] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:45.014194 2468 leader_election.cc:290] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:45.014362 1883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4054e1b6e654bc4b57d42be2a85296e" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
I20250110 03:07:45.014371 1751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4054e1b6e654bc4b57d42be2a85296e" candidate_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:45.014425 1883 raft_consensus.cc:3054] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:45.014441 1751 raft_consensus.cc:3054] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:45.015004 1751 raft_consensus.cc:2463] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:45.015065 1883 raft_consensus.cc:2463] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baa5f3e8148b4fdda964dbc1cb4a30c0 in term 1.
I20250110 03:07:45.015199 1553 leader_election.cc:304] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:45.015309 2468 raft_consensus.cc:2798] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:45.015376 2468 raft_consensus.cc:695] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Leader. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:45.015429 2468 consensus_queue.cc:237] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } }
I20250110 03:07:45.015903 1468 catalog_manager.cc:5526] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:45.103066 2487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:45.103224 2487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:45.104496 2487 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20250110 03:07:45.132401 1883 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:45.132401 1882 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:45.132584 1881 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:45.132632 1751 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:45.132633 1750 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:45.132771 1748 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 03:07:45.132910 2365 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:45.132915 2354 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:45.133015 2468 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:45.133070 2354 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:45.133282 2512 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:45.133381 2516 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 19.2453 ms
time per row: 0.00962266 ms
I20250110 03:07:45.147219 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.134:38075
--local_ip_for_outbound_sockets=127.14.19.134
--tserver_master_addrs=127.14.19.190:44425
--webserver_port=44523
--webserver_interface=127.14.19.134
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:45.218493 2529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:45.218684 2529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:45.218704 2529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:45.219888 2529 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:45.219935 2529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:45.219957 2529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.134
I20250110 03:07:45.221275 2529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.134:38075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.14.19.134
--webserver_port=44523
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.2529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.134
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:45.221444 2529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:45.221657 2529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250110 03:07:45.226498 2529 server_base.cc:1034] running on GCE node
W20250110 03:07:45.227190 2537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:45.227553 2534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:45.228472 2535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:45.228626 2529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:45.228816 2529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:45.229940 2529 hybrid_clock.cc:648] HybridClock initialized: now 1736478465229919 us; error 34 us; skew 500 ppm
I20250110 03:07:45.230976 2529 webserver.cc:458] Webserver started at http://127.14.19.134:44523/ using document root <none> and password file <none>
I20250110 03:07:45.231146 2529 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:45.231210 2529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:45.232308 2529 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:45.232771 2543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.232950 2529 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.233013 2529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:45.233235 2529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:45.242784 2529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:45.242944 2529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:45.243037 2529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:45.243184 2529 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:45.243415 2529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:45.243448 2529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.243476 2529 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:45.243507 2529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.248627 2529 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.134:38075
I20250110 03:07:45.248659 2656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.134:38075 every 8 connection(s)
I20250110 03:07:45.248912 2529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20250110 03:07:45.251173 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 2529
I20250110 03:07:45.251406 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:36633
--local_ip_for_outbound_sockets=127.14.19.133
--tserver_master_addrs=127.14.19.190:44425
--webserver_port=41981
--webserver_interface=127.14.19.133
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:45.253252 2657 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:45.253327 2657 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:45.253481 2657 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:45.253847 1470 ts_manager.cc:194] Re-registered known tserver with Master: 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075)
I20250110 03:07:45.254360 1470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.134:46093
W20250110 03:07:45.323633 2660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:45.323768 2660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:45.323786 2660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:45.325049 2660 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:45.325099 2660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:45.325131 2660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:07:45.326524 2660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:36633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=41981
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.2660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:45.326738 2660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:45.326939 2660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:45.330657 2665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:45.331648 2666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:45.332751 2668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:45.333670 2660 server_base.cc:1034] running on GCE node
I20250110 03:07:45.333799 2660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:45.333962 2660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:45.335093 2660 hybrid_clock.cc:648] HybridClock initialized: now 1736478465335069 us; error 35 us; skew 500 ppm
I20250110 03:07:45.336156 2660 webserver.cc:458] Webserver started at http://127.14.19.133:41981/ using document root <none> and password file <none>
I20250110 03:07:45.336338 2660 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:45.336398 2660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:45.337527 2660 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:45.337985 2674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.338128 2660 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.338191 2660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c2867695ca20469dba7d61ebc5ecebfa"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:45.338423 2660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:45.347189 2660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:45.347364 2660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:45.347460 2660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:45.347611 2660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:45.347841 2660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:45.347873 2660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.347900 2660 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:45.347930 2660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.353010 2660 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:36633
I20250110 03:07:45.353044 2787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:36633 every 8 connection(s)
I20250110 03:07:45.353302 2660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:07:45.355649 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 2660
I20250110 03:07:45.355881 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:40673
--local_ip_for_outbound_sockets=127.14.19.132
--tserver_master_addrs=127.14.19.190:44425
--webserver_port=45285
--webserver_interface=127.14.19.132
--builtin_ntp_servers=127.14.19.148:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:45.357568 2788 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:45.357654 2788 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:45.357836 2788 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:45.358186 1470 ts_manager.cc:194] Re-registered known tserver with Master: c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633)
I20250110 03:07:45.358718 1470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:56871
W20250110 03:07:45.429898 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:45.430042 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:45.430061 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:45.431324 2791 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:45.431363 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:45.431386 2791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:45.432842 2791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:40673
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=45285
--tserver_master_addrs=127.14.19.190:44425
--never_fsync=true
--heap_profile_path=/tmp/kudu.2791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:45.433022 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:45.433184 2791 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:45.436885 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:45.437805 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:45.438634 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:45.439463 2791 server_base.cc:1034] running on GCE node
I20250110 03:07:45.439617 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:45.439800 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:45.440922 2791 hybrid_clock.cc:648] HybridClock initialized: now 1736478465440911 us; error 26 us; skew 500 ppm
I20250110 03:07:45.441954 2791 webserver.cc:458] Webserver started at http://127.14.19.132:45285/ using document root <none> and password file <none>
I20250110 03:07:45.442124 2791 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:45.442173 2791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:45.443235 2791 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:45.443672 2805 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.443817 2791 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.443876 2791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "856267e243cb46ab884b7a1694ae465e"
format_stamp: "Formatted at 2025-01-10 03:07:39 on dist-test-slave-nrvn"
I20250110 03:07:45.444116 2791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:45.459437 2791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:45.459625 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:45.459720 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:45.459883 2791 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:45.460129 2791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:45.460158 2791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.460189 2791 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:45.460213 2791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:45.465376 2791 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:40673
I20250110 03:07:45.465410 2918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:40673 every 8 connection(s)
I20250110 03:07:45.465677 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:45.469619 2919 heartbeater.cc:346] Connected to a master server at 127.14.19.190:44425
I20250110 03:07:45.469691 2919 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:45.469848 2919 heartbeater.cc:510] Master 127.14.19.190:44425 requested a full tablet report, sending...
I20250110 03:07:45.470149 1470 ts_manager.cc:194] Re-registered known tserver with Master: 856267e243cb46ab884b7a1694ae465e (127.14.19.132:40673)
I20250110 03:07:45.470336 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 2791
I20250110 03:07:45.470660 1470 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:54075
I20250110 03:07:45.586243 1863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:45.586840 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:45.596359 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.596475 1599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:45.597368 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.607564 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.645120 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:45.647743 1599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:45.648029 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.649668 1863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:45.649933 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.658589 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.701508 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:45.702808 1599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:45.704878 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.707872 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.711047 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.711300 1863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:45.744593 1863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20250110 03:07:45.744748 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.745512 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:45.747216 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.749445 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:45.753168 1599 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20250110 03:07:45.763535 1619 consensus_queue.cc:237] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:45.764081 1750 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.764240 1882 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.764429 3073 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.764551 3072 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.765331 2514 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.765779 1882 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.766459 1750 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.766782 1470 catalog_manager.cc:5526] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:45.767279 1552 consensus_peers.cc:487] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 856267e243cb46ab884b7a1694ae465e (127.14.19.132:40673): Couldn't send request to peer 856267e243cb46ab884b7a1694ae465e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a02da6c3e2d4db5b1db2f1de6824bc5. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.771174 1882 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:45.771625 1750 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.771657 1619 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.771839 3085 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.771972 3086 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.772461 3086 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.772569 1750 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.772745 1619 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.773533 1473 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
W20250110 03:07:45.775905 1819 consensus_peers.cc:487] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 -> Peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075): Couldn't send request to peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48cdbe84b6094cccaf69b553fd8f1d68. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.777366 1619 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:45.777709 1750 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.777711 1882 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.777881 3073 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.778017 2514 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.778671 2514 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.778769 1882 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.779636 1750 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.779649 1474 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
W20250110 03:07:45.781133 1555 consensus_peers.cc:487] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633): Couldn't send request to peer c2867695ca20469dba7d61ebc5ecebfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eefff01740a34c40ba687c7242b87770. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.784006 1619 consensus_queue.cc:237] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:45.784368 1882 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.784397 1750 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:45.784443 1552 consensus_peers.cc:487] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 856267e243cb46ab884b7a1694ae465e (127.14.19.132:40673): Couldn't send request to peer 856267e243cb46ab884b7a1694ae465e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4054e1b6e654bc4b57d42be2a85296e. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.784575 3072 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.784657 3073 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.785151 2514 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.785274 1882 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.785336 1750 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.786310 1473 catalog_manager.cc:5526] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: config changed from index -1 to 4, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:45.789726 1619 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:45.790095 1750 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.790109 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.790278 2514 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.790431 2514 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.790815 3072 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.790975 1882 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.791110 1750 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.791895 1474 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:45.793416 1555 consensus_peers.cc:487] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075): Couldn't send request to peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d68741aa20be48229a19dda143743bca. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.796082 1750 consensus_queue.cc:237] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:45.796593 1882 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.796780 1619 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.796824 3110 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.796947 3111 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.797447 3110 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.797566 1882 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.798586 1473 catalog_manager.cc:5526] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "90d44e46d37f48258cea315fd7e589d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.798988 1619 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
W20250110 03:07:45.800412 1687 consensus_peers.cc:487] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 -> Peer c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633): Couldn't send request to peer c2867695ca20469dba7d61ebc5ecebfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b373739a50754ecb8753a84d813c0151. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.802425 1619 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:45.802853 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.802879 1750 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:45.802927 1555 consensus_peers.cc:487] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075): Couldn't send request to peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56aea05a8a714b4ca357e24049c7f348. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.803018 3073 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.803090 3072 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.803614 3072 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.803750 1750 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.803743 1882 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:45.804983 1474 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index -1 to 4, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:45.808370 1619 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:45.808723 1750 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:45.808764 1882 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:45.808904 1555 consensus_peers.cc:487] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633): Couldn't send request to peer c2867695ca20469dba7d61ebc5ecebfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0eee4c5c9b54dfca9b59276787773db. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:45.808954 2514 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.809026 3073 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:45.809558 3072 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.809666 1750 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.809681 1882 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:45.810528 1470 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index -1 to 4, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:46.181888 3205 ts_tablet_manager.cc:927] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:46.182097 3205 tablet_copy_client.cc:323] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:46.183104 3208 ts_tablet_manager.cc:927] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:46.183292 3208 tablet_copy_client.cc:323] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:46.185340 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet eefff01740a34c40ba687c7242b87770 from peer c2867695ca20469dba7d61ebc5ecebfa ({username='slave'} at 127.14.19.133:45019)
I20250110 03:07:46.185417 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet eefff01740a34c40ba687c7242b87770 from peer c2867695ca20469dba7d61ebc5ecebfa at {username='slave'} at 127.14.19.133:45019: session id = c2867695ca20469dba7d61ebc5ecebfa-eefff01740a34c40ba687c7242b87770
I20250110 03:07:46.186094 1639 tablet_copy_source_session.cc:215] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.186342 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet 56aea05a8a714b4ca357e24049c7f348 from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec ({username='slave'} at 127.14.19.134:50413)
I20250110 03:07:46.186399 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet 56aea05a8a714b4ca357e24049c7f348 from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec at {username='slave'} at 127.14.19.134:50413: session id = 9ddba97e3a4f4b4b99e67a34c4a0e3ec-56aea05a8a714b4ca357e24049c7f348
I20250110 03:07:46.186808 1639 tablet_copy_source_session.cc:215] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.186902 3205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefff01740a34c40ba687c7242b87770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.187364 3208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aea05a8a714b4ca357e24049c7f348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.189030 3205 tablet_copy_client.cc:806] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.189167 3205 tablet_copy_client.cc:670] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.189433 3208 tablet_copy_client.cc:806] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.189569 3208 tablet_copy_client.cc:670] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.190011 3205 tablet_copy_client.cc:538] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.190249 3208 tablet_copy_client.cc:538] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.190649 3205 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap starting.
I20250110 03:07:46.190860 3208 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap starting.
I20250110 03:07:46.192344 3212 ts_tablet_manager.cc:927] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:46.192395 3205 log.cc:826] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:46.192584 3212 tablet_copy_client.cc:323] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:46.192636 3208 log.cc:826] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:46.192775 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet d68741aa20be48229a19dda143743bca from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec ({username='slave'} at 127.14.19.134:50413)
I20250110 03:07:46.192833 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet d68741aa20be48229a19dda143743bca from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec at {username='slave'} at 127.14.19.134:50413: session id = 9ddba97e3a4f4b4b99e67a34c4a0e3ec-d68741aa20be48229a19dda143743bca
I20250110 03:07:46.193300 1639 tablet_copy_source_session.cc:215] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.193567 3212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d68741aa20be48229a19dda143743bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.194342 3212 tablet_copy_client.cc:806] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.194459 3212 tablet_copy_client.cc:670] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.195108 3212 tablet_copy_client.cc:538] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.195752 3212 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap starting.
I20250110 03:07:46.196553 3205 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.196834 3205 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap complete.
I20250110 03:07:46.196956 3205 ts_tablet_manager.cc:1397] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250110 03:07:46.197382 3208 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.197598 3208 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap complete.
I20250110 03:07:46.197705 3208 ts_tablet_manager.cc:1397] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250110 03:07:46.198087 3205 raft_consensus.cc:357] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:46.198248 3205 raft_consensus.cc:738] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2867695ca20469dba7d61ebc5ecebfa, State: Initialized, Role: LEARNER
I20250110 03:07:46.198316 3205 consensus_queue.cc:260] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:46.198559 3205 ts_tablet_manager.cc:1428] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250110 03:07:46.198596 2788 heartbeater.cc:502] Master 127.14.19.190:44425 was elected leader, sending a full tablet report...
I20250110 03:07:46.198511 3208 raft_consensus.cc:357] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:46.198659 3208 raft_consensus.cc:738] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ddba97e3a4f4b4b99e67a34c4a0e3ec, State: Initialized, Role: LEARNER
I20250110 03:07:46.198714 3208 consensus_queue.cc:260] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:46.198771 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session c2867695ca20469dba7d61ebc5ecebfa-eefff01740a34c40ba687c7242b87770 received from {username='slave'} at 127.14.19.133:45019
I20250110 03:07:46.198817 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session c2867695ca20469dba7d61ebc5ecebfa-eefff01740a34c40ba687c7242b87770 on tablet eefff01740a34c40ba687c7242b87770 with peer c2867695ca20469dba7d61ebc5ecebfa
I20250110 03:07:46.198910 3208 ts_tablet_manager.cc:1428] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20250110 03:07:46.199043 2657 heartbeater.cc:502] Master 127.14.19.190:44425 was elected leader, sending a full tablet report...
I20250110 03:07:46.199429 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-56aea05a8a714b4ca357e24049c7f348 received from {username='slave'} at 127.14.19.134:50413
I20250110 03:07:46.199476 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-56aea05a8a714b4ca357e24049c7f348 on tablet 56aea05a8a714b4ca357e24049c7f348 with peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec
I20250110 03:07:46.200502 3212 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.200649 3212 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap complete.
I20250110 03:07:46.200716 3212 ts_tablet_manager.cc:1397] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250110 03:07:46.200793 3212 raft_consensus.cc:357] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:46.200838 3212 raft_consensus.cc:738] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ddba97e3a4f4b4b99e67a34c4a0e3ec, State: Initialized, Role: LEARNER
I20250110 03:07:46.200858 3212 consensus_queue.cc:260] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:46.200917 3212 ts_tablet_manager.cc:1428] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:46.201166 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-d68741aa20be48229a19dda143743bca received from {username='slave'} at 127.14.19.134:50413
I20250110 03:07:46.201212 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-d68741aa20be48229a19dda143743bca on tablet d68741aa20be48229a19dda143743bca with peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec
I20250110 03:07:46.293485 3259 ts_tablet_manager.cc:927] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:46.293649 3259 tablet_copy_client.cc:323] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:46.295738 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 from peer 856267e243cb46ab884b7a1694ae465e ({username='slave'} at 127.14.19.132:37257)
I20250110 03:07:46.295799 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 from peer 856267e243cb46ab884b7a1694ae465e at {username='slave'} at 127.14.19.132:37257: session id = 856267e243cb46ab884b7a1694ae465e-0a02da6c3e2d4db5b1db2f1de6824bc5
I20250110 03:07:46.296175 1639 tablet_copy_source_session.cc:215] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.296799 3259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.298879 3259 tablet_copy_client.cc:806] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.299012 3259 tablet_copy_client.cc:670] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.299655 3259 tablet_copy_client.cc:538] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.300249 3259 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Bootstrap starting.
I20250110 03:07:46.301630 3259 log.cc:826] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:46.304884 3259 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.305078 3259 tablet_bootstrap.cc:492] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Bootstrap complete.
I20250110 03:07:46.305181 3259 ts_tablet_manager.cc:1397] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250110 03:07:46.305876 3259 raft_consensus.cc:357] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:46.305991 3259 raft_consensus.cc:738] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 856267e243cb46ab884b7a1694ae465e, State: Initialized, Role: LEARNER
I20250110 03:07:46.306054 3259 consensus_queue.cc:260] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:46.306248 3259 ts_tablet_manager.cc:1428] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250110 03:07:46.306295 2919 heartbeater.cc:502] Master 127.14.19.190:44425 was elected leader, sending a full tablet report...
I20250110 03:07:46.306469 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session 856267e243cb46ab884b7a1694ae465e-0a02da6c3e2d4db5b1db2f1de6824bc5 received from {username='slave'} at 127.14.19.132:37257
I20250110 03:07:46.306526 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session 856267e243cb46ab884b7a1694ae465e-0a02da6c3e2d4db5b1db2f1de6824bc5 on tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 with peer 856267e243cb46ab884b7a1694ae465e
I20250110 03:07:46.332558 3212 ts_tablet_manager.cc:927] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Initiating tablet copy from peer 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:46.332696 3212 tablet_copy_client.cc:323] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:42095
I20250110 03:07:46.334969 1903 tablet_copy_service.cc:140] P 84c4433aa11f4e2f9a49bdb01aba84e0: Received BeginTabletCopySession request for tablet 48cdbe84b6094cccaf69b553fd8f1d68 from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec ({username='slave'} at 127.14.19.134:43625)
I20250110 03:07:46.335033 1903 tablet_copy_service.cc:161] P 84c4433aa11f4e2f9a49bdb01aba84e0: Beginning new tablet copy session on tablet 48cdbe84b6094cccaf69b553fd8f1d68 from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec at {username='slave'} at 127.14.19.134:43625: session id = 9ddba97e3a4f4b4b99e67a34c4a0e3ec-48cdbe84b6094cccaf69b553fd8f1d68
I20250110 03:07:46.335484 1903 tablet_copy_source_session.cc:215] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.335853 3212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cdbe84b6094cccaf69b553fd8f1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.336778 3212 tablet_copy_client.cc:806] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.336884 3212 tablet_copy_client.cc:670] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.337541 3212 tablet_copy_client.cc:538] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.338074 3212 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap starting.
I20250110 03:07:46.340756 3212 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.340955 3212 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap complete.
I20250110 03:07:46.341044 3212 ts_tablet_manager.cc:1397] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:46.341159 3212 raft_consensus.cc:357] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:46.341223 3212 raft_consensus.cc:738] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ddba97e3a4f4b4b99e67a34c4a0e3ec, State: Initialized, Role: LEARNER
I20250110 03:07:46.341259 3212 consensus_queue.cc:260] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:46.341357 3212 ts_tablet_manager.cc:1428] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:46.341571 1903 tablet_copy_service.cc:342] P 84c4433aa11f4e2f9a49bdb01aba84e0: Request end of tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-48cdbe84b6094cccaf69b553fd8f1d68 received from {username='slave'} at 127.14.19.134:43625
I20250110 03:07:46.341627 1903 tablet_copy_service.cc:434] P 84c4433aa11f4e2f9a49bdb01aba84e0: ending tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-48cdbe84b6094cccaf69b553fd8f1d68 on tablet 48cdbe84b6094cccaf69b553fd8f1d68 with peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec
I20250110 03:07:46.393764 3205 ts_tablet_manager.cc:927] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:46.393918 3205 tablet_copy_client.cc:323] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:46.394134 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet d0eee4c5c9b54dfca9b59276787773db from peer c2867695ca20469dba7d61ebc5ecebfa ({username='slave'} at 127.14.19.133:45019)
I20250110 03:07:46.394191 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet d0eee4c5c9b54dfca9b59276787773db from peer c2867695ca20469dba7d61ebc5ecebfa at {username='slave'} at 127.14.19.133:45019: session id = c2867695ca20469dba7d61ebc5ecebfa-d0eee4c5c9b54dfca9b59276787773db
I20250110 03:07:46.394532 1639 tablet_copy_source_session.cc:215] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.394876 3205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0eee4c5c9b54dfca9b59276787773db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.395659 3205 tablet_copy_client.cc:806] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.395779 3205 tablet_copy_client.cc:670] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.396323 3205 tablet_copy_client.cc:538] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.396833 3205 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap starting.
I20250110 03:07:46.399474 3205 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.399652 3205 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap complete.
I20250110 03:07:46.399735 3205 ts_tablet_manager.cc:1397] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:46.399839 3205 raft_consensus.cc:357] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:46.399899 3205 raft_consensus.cc:738] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2867695ca20469dba7d61ebc5ecebfa, State: Initialized, Role: LEARNER
I20250110 03:07:46.399937 3205 consensus_queue.cc:260] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:46.400030 3205 ts_tablet_manager.cc:1428] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:46.400229 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session c2867695ca20469dba7d61ebc5ecebfa-d0eee4c5c9b54dfca9b59276787773db received from {username='slave'} at 127.14.19.133:45019
I20250110 03:07:46.400285 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session c2867695ca20469dba7d61ebc5ecebfa-d0eee4c5c9b54dfca9b59276787773db on tablet d0eee4c5c9b54dfca9b59276787773db with peer c2867695ca20469dba7d61ebc5ecebfa
I20250110 03:07:46.401119 3259 ts_tablet_manager.cc:927] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:46.401234 3259 tablet_copy_client.cc:323] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:46.401412 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet a4054e1b6e654bc4b57d42be2a85296e from peer 856267e243cb46ab884b7a1694ae465e ({username='slave'} at 127.14.19.132:37257)
I20250110 03:07:46.401464 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet a4054e1b6e654bc4b57d42be2a85296e from peer 856267e243cb46ab884b7a1694ae465e at {username='slave'} at 127.14.19.132:37257: session id = 856267e243cb46ab884b7a1694ae465e-a4054e1b6e654bc4b57d42be2a85296e
I20250110 03:07:46.401921 1639 tablet_copy_source_session.cc:215] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.402216 3259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4054e1b6e654bc4b57d42be2a85296e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.403065 3259 tablet_copy_client.cc:806] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.403177 3259 tablet_copy_client.cc:670] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.403688 3259 tablet_copy_client.cc:538] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.404135 3259 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: Bootstrap starting.
I20250110 03:07:46.406791 3259 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.406944 3259 tablet_bootstrap.cc:492] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: Bootstrap complete.
I20250110 03:07:46.407016 3259 ts_tablet_manager.cc:1397] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20250110 03:07:46.407114 3259 raft_consensus.cc:357] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:46.407166 3259 raft_consensus.cc:738] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 856267e243cb46ab884b7a1694ae465e, State: Initialized, Role: LEARNER
I20250110 03:07:46.407186 3259 consensus_queue.cc:260] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:46.407255 3259 ts_tablet_manager.cc:1428] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:46.407452 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session 856267e243cb46ab884b7a1694ae465e-a4054e1b6e654bc4b57d42be2a85296e received from {username='slave'} at 127.14.19.132:37257
I20250110 03:07:46.407501 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session 856267e243cb46ab884b7a1694ae465e-a4054e1b6e654bc4b57d42be2a85296e on tablet a4054e1b6e654bc4b57d42be2a85296e with peer 856267e243cb46ab884b7a1694ae465e
I20250110 03:07:46.419818 3205 ts_tablet_manager.cc:927] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: Initiating tablet copy from peer 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231)
I20250110 03:07:46.419934 3205 tablet_copy_client.cc:323] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:38231
I20250110 03:07:46.422115 1771 tablet_copy_service.cc:140] P 90d44e46d37f48258cea315fd7e589d5: Received BeginTabletCopySession request for tablet b373739a50754ecb8753a84d813c0151 from peer c2867695ca20469dba7d61ebc5ecebfa ({username='slave'} at 127.14.19.133:51435)
I20250110 03:07:46.422180 1771 tablet_copy_service.cc:161] P 90d44e46d37f48258cea315fd7e589d5: Beginning new tablet copy session on tablet b373739a50754ecb8753a84d813c0151 from peer c2867695ca20469dba7d61ebc5ecebfa at {username='slave'} at 127.14.19.133:51435: session id = c2867695ca20469dba7d61ebc5ecebfa-b373739a50754ecb8753a84d813c0151
I20250110 03:07:46.422689 1771 tablet_copy_source_session.cc:215] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:46.423074 3205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b373739a50754ecb8753a84d813c0151. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:46.423906 3205 tablet_copy_client.cc:806] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:46.424023 3205 tablet_copy_client.cc:670] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:46.424670 3205 tablet_copy_client.cc:538] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:46.425159 3205 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap starting.
I20250110 03:07:46.427623 3205 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:46.427770 3205 tablet_bootstrap.cc:492] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap complete.
I20250110 03:07:46.427839 3205 ts_tablet_manager.cc:1397] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:46.427923 3205 raft_consensus.cc:357] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:46.427982 3205 raft_consensus.cc:738] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2867695ca20469dba7d61ebc5ecebfa, State: Initialized, Role: LEARNER
I20250110 03:07:46.428014 3205 consensus_queue.cc:260] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:46.428099 3205 ts_tablet_manager.cc:1428] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:46.428361 1771 tablet_copy_service.cc:342] P 90d44e46d37f48258cea315fd7e589d5: Request end of tablet copy session c2867695ca20469dba7d61ebc5ecebfa-b373739a50754ecb8753a84d813c0151 received from {username='slave'} at 127.14.19.133:51435
I20250110 03:07:46.428417 1771 tablet_copy_service.cc:434] P 90d44e46d37f48258cea315fd7e589d5: ending tablet copy session c2867695ca20469dba7d61ebc5ecebfa-b373739a50754ecb8753a84d813c0151 on tablet b373739a50754ecb8753a84d813c0151 with peer c2867695ca20469dba7d61ebc5ecebfa
I20250110 03:07:46.701020 2873 raft_consensus.cc:1212] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.715320 2742 raft_consensus.cc:1212] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.765055 2611 raft_consensus.cc:1212] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.780349 2611 raft_consensus.cc:1212] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.798141 2742 raft_consensus.cc:1212] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.802397 2611 raft_consensus.cc:1212] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.877892 2742 raft_consensus.cc:1212] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:46.946683 2873 raft_consensus.cc:1212] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:47.146692 3072 raft_consensus.cc:1059] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec to VOTER
I20250110 03:07:47.146908 3072 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:47.147456 1750 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.147455 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.147475 2611 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.147667 3397 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.147853 3072 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.148032 3266 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.148734 3072 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.148864 1750 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.149152 2611 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.149859 1882 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.150106 1473 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.151522 1619 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:47.151865 2611 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.151888 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.152050 3397 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.152123 3318 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.152699 3266 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.152827 1882 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.153795 1474 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.154117 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 56aea05a8a714b4ca357e24049c7f348 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:47.154266 2611 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.155197 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet 56aea05a8a714b4ca357e24049c7f348 with delete_type TABLET_DATA_TOMBSTONED (TS 90d44e46d37f48258cea315fd7e589d5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56738
I20250110 03:07:47.155436 3405 tablet_replica.cc:331] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: stopping tablet replica
I20250110 03:07:47.155534 3405 raft_consensus.cc:2238] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:47.155613 3405 raft_consensus.cc:2267] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:47.155902 3405 ts_tablet_manager.cc:1905] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:47.156967 3405 ts_tablet_manager.cc:1918] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:47.157028 3405 log.cc:1198] T 56aea05a8a714b4ca357e24049c7f348 P 90d44e46d37f48258cea315fd7e589d5: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/56aea05a8a714b4ca357e24049c7f348
I20250110 03:07:47.157409 1462 catalog_manager.cc:4872] TS 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): tablet 56aea05a8a714b4ca357e24049c7f348 (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:47.197379 3354 raft_consensus.cc:1059] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: attempting to promote NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec to VOTER
I20250110 03:07:47.197592 3354 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:47.198107 1619 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.198093 1750 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.198191 2611 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.198293 3085 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.198400 3354 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.198573 3354 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.199488 3463 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.199712 1750 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.199716 1619 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.200783 1470 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:47.201081 2611 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.201822 1882 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:47.202634 1619 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.202803 2611 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.202862 3085 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.202977 3085 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.203423 3463 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.203583 2611 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.203608 1619 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.204581 1474 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec reported cstate change: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.204699 1461 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 48cdbe84b6094cccaf69b553fd8f1d68 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:47.205430 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cdbe84b6094cccaf69b553fd8f1d68 with delete_type TABLET_DATA_TOMBSTONED (TS 90d44e46d37f48258cea315fd7e589d5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56738
I20250110 03:07:47.205529 3405 tablet_replica.cc:331] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: stopping tablet replica
I20250110 03:07:47.205600 3405 raft_consensus.cc:2238] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:47.205662 3405 raft_consensus.cc:2267] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:47.205910 3405 ts_tablet_manager.cc:1905] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:47.207134 3405 ts_tablet_manager.cc:1918] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:47.207187 3405 log.cc:1198] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/48cdbe84b6094cccaf69b553fd8f1d68
I20250110 03:07:47.207504 1462 catalog_manager.cc:4872] TS 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): tablet 48cdbe84b6094cccaf69b553fd8f1d68 (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:47.288339 3318 raft_consensus.cc:1059] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER 856267e243cb46ab884b7a1694ae465e to VOTER
I20250110 03:07:47.288542 3318 consensus_queue.cc:237] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:47.289047 2873 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.289033 1750 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.289067 1882 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.289237 3266 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.289335 3397 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.289461 3318 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.290222 3266 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.290360 1882 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.290517 2873 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.291404 1750 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.291523 1474 catalog_manager.cc:5526] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:47.292624 1619 consensus_queue.cc:237] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:47.292955 2873 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.292979 1750 raft_consensus.cc:1270] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.293147 3318 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.293216 3397 consensus_queue.cc:1035] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.293828 3397 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.293947 2873 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.294032 1750 raft_consensus.cc:2949] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.294776 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:47.294831 1470 catalog_manager.cc:5526] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 856267e243cb46ab884b7a1694ae465e reported cstate change: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.295501 3318 raft_consensus.cc:1059] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER c2867695ca20469dba7d61ebc5ecebfa to VOTER
I20250110 03:07:47.295627 3318 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.295622 1863 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 with delete_type TABLET_DATA_TOMBSTONED (TS 84c4433aa11f4e2f9a49bdb01aba84e0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54160
I20250110 03:07:47.295840 3491 tablet_replica.cc:331] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: stopping tablet replica
I20250110 03:07:47.296034 1750 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.296069 2742 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.296206 3491 raft_consensus.cc:2238] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:47.296186 3266 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.296226 1882 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.296310 3491 raft_consensus.cc:2267] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:47.296299 3266 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.296463 3072 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.296667 3491 ts_tablet_manager.cc:1905] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:47.297255 3266 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.297467 2742 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.298115 3491 ts_tablet_manager.cc:1918] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:47.298187 3491 log.cc:1198] T 0a02da6c3e2d4db5b1db2f1de6824bc5 P 84c4433aa11f4e2f9a49bdb01aba84e0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/0a02da6c3e2d4db5b1db2f1de6824bc5
I20250110 03:07:47.298209 1750 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.298249 1472 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:47.298626 1461 catalog_manager.cc:4872] TS 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095): tablet 0a02da6c3e2d4db5b1db2f1de6824bc5 (table rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]) successfully deleted
I20250110 03:07:47.299029 1882 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.304598 3318 raft_consensus.cc:1059] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER c2867695ca20469dba7d61ebc5ecebfa to VOTER
I20250110 03:07:47.304726 3318 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.305135 1750 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.305135 1882 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.305233 2742 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.305342 3397 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.305423 3266 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.305501 3072 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.306092 3397 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.306212 2742 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.306386 1750 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.306988 1882 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.307289 1472 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa reported cstate change: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.405647 3072 raft_consensus.cc:1059] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec to VOTER
I20250110 03:07:47.405830 3072 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:47.406351 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.406355 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.406355 1750 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.406592 3072 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.406692 3318 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.406837 3397 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.407500 3266 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.407609 1750 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.407665 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.408416 1882 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.408736 1473 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 4 to 5, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:47.410076 1619 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:47.410463 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.410470 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.410708 3318 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.410791 3397 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.411361 3266 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.411486 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.412333 1882 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:47.412400 1470 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 5 to 6, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:47.412607 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d68741aa20be48229a19dda143743bca with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:47.413141 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet d68741aa20be48229a19dda143743bca with delete_type TABLET_DATA_TOMBSTONED (TS 90d44e46d37f48258cea315fd7e589d5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56738
I20250110 03:07:47.413249 3405 tablet_replica.cc:331] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: stopping tablet replica
I20250110 03:07:47.413305 3405 raft_consensus.cc:2238] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:47.413372 3405 raft_consensus.cc:2267] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:47.413620 3405 ts_tablet_manager.cc:1905] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:47.414702 3405 ts_tablet_manager.cc:1918] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:47.414749 3405 log.cc:1198] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/d68741aa20be48229a19dda143743bca
I20250110 03:07:47.415061 1462 catalog_manager.cc:4872] TS 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): tablet d68741aa20be48229a19dda143743bca (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:47.427703 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eefff01740a34c40ba687c7242b87770"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50340
I20250110 03:07:47.427774 1619 raft_consensus.cc:604] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.433691 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50358
I20250110 03:07:47.433741 1619 raft_consensus.cc:604] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.451833 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eefff01740a34c40ba687c7242b87770"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50394
I20250110 03:07:47.451907 1619 raft_consensus.cc:604] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.457773 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50420
I20250110 03:07:47.457828 1619 raft_consensus.cc:604] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.465822 3579 raft_consensus.cc:1059] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5: attempting to promote NON_VOTER c2867695ca20469dba7d61ebc5ecebfa to VOTER
I20250110 03:07:47.466019 3579 consensus_queue.cc:237] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.466579 2742 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.466614 1882 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.466636 1619 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.466794 3582 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.466950 3579 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.467032 3110 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.467854 3110 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.468063 2742 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.468724 1882 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.468823 1619 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.468957 1470 catalog_manager.cc:5526] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa reported cstate change: config changed from index 4 to 5, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "90d44e46d37f48258cea315fd7e589d5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.469954 1750 consensus_queue.cc:237] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.470369 1882 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.470379 2742 raft_consensus.cc:1270] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Refusing update from remote peer 90d44e46d37f48258cea315fd7e589d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.470611 3581 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.470723 3582 consensus_queue.cc:1035] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.471299 3581 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.471422 1882 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.471603 2742 raft_consensus.cc:2949] T b373739a50754ecb8753a84d813c0151 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.472285 1462 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b373739a50754ecb8753a84d813c0151 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:47.472527 1472 catalog_manager.cc:5526] T b373739a50754ecb8753a84d813c0151 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 5 to 6, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "90d44e46d37f48258cea315fd7e589d5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:47.473197 1599 tablet_service.cc:1514] Processing DeleteTablet for tablet b373739a50754ecb8753a84d813c0151 with delete_type TABLET_DATA_TOMBSTONED (TS baa5f3e8148b4fdda964dbc1cb4a30c0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49710
I20250110 03:07:47.473371 3586 tablet_replica.cc:331] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: stopping tablet replica
I20250110 03:07:47.473452 3586 raft_consensus.cc:2238] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:47.473503 3586 raft_consensus.cc:2267] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:47.473737 3586 ts_tablet_manager.cc:1905] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:47.474735 3586 ts_tablet_manager.cc:1918] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:47.474786 3586 log.cc:1198] T b373739a50754ecb8753a84d813c0151 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/b373739a50754ecb8753a84d813c0151
I20250110 03:07:47.475132 1463 catalog_manager.cc:4872] TS baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179): tablet b373739a50754ecb8753a84d813c0151 (table rebalance_test_table_0 [id=b8710a737ffd4f5f93b9e9df5862bb05]) successfully deleted
I20250110 03:07:47.502928 3397 raft_consensus.cc:1059] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER 856267e243cb46ab884b7a1694ae465e to VOTER
I20250110 03:07:47.503070 3397 consensus_queue.cc:237] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:47.503517 1750 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.503521 2873 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.503546 1882 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:47.503691 3318 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.503775 3266 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.503877 3072 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:47.504674 3318 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.504804 1750 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.505741 1882 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.505970 1473 catalog_manager.cc:5526] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.506270 2873 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.507387 1619 consensus_queue.cc:237] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:47.507719 1750 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.507719 2873 raft_consensus.cc:1270] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:47.507879 3266 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.507972 3397 consensus_queue.cc:1035] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:47.508464 3318 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.508582 2873 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.509275 1750 raft_consensus.cc:2949] T a4054e1b6e654bc4b57d42be2a85296e P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.509425 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a4054e1b6e654bc4b57d42be2a85296e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:47.509630 1470 catalog_manager.cc:5526] T a4054e1b6e654bc4b57d42be2a85296e P 856267e243cb46ab884b7a1694ae465e reported cstate change: config changed from index 5 to 6, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:47.510473 1863 tablet_service.cc:1514] Processing DeleteTablet for tablet a4054e1b6e654bc4b57d42be2a85296e with delete_type TABLET_DATA_TOMBSTONED (TS 84c4433aa11f4e2f9a49bdb01aba84e0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54160
I20250110 03:07:47.510612 3491 tablet_replica.cc:331] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: stopping tablet replica
I20250110 03:07:47.510685 3491 raft_consensus.cc:2238] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:47.510746 3491 raft_consensus.cc:2267] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:47.511034 3491 ts_tablet_manager.cc:1905] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:47.512076 3491 ts_tablet_manager.cc:1918] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:47.512122 3491 log.cc:1198] T a4054e1b6e654bc4b57d42be2a85296e P 84c4433aa11f4e2f9a49bdb01aba84e0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/a4054e1b6e654bc4b57d42be2a85296e
I20250110 03:07:47.512444 1461 catalog_manager.cc:4872] TS 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095): tablet a4054e1b6e654bc4b57d42be2a85296e (table rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]) successfully deleted
I20250110 03:07:47.670759 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eefff01740a34c40ba687c7242b87770"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50446
I20250110 03:07:47.670847 1619 raft_consensus.cc:604] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.676709 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50454
I20250110 03:07:47.676769 1619 raft_consensus.cc:604] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.688809 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eefff01740a34c40ba687c7242b87770"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50492
I20250110 03:07:47.688864 1619 raft_consensus.cc:604] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.694602 1619 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db"
dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50504
I20250110 03:07:47.694669 1619 raft_consensus.cc:604] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:07:47.757557 3318 raft_consensus.cc:988] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: : Instructing follower 84c4433aa11f4e2f9a49bdb01aba84e0 to start an election
I20250110 03:07:47.757642 3318 raft_consensus.cc:1076] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Signalling peer 84c4433aa11f4e2f9a49bdb01aba84e0 to start an election
I20250110 03:07:47.757874 1882 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eefff01740a34c40ba687c7242b87770"
dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
from {username='slave'} at 127.14.19.129:46789
I20250110 03:07:47.757957 1882 raft_consensus.cc:491] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:47.757979 1882 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.758585 1882 raft_consensus.cc:513] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.758759 1882 leader_election.cc:290] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 2 election: Requested vote from peers baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179), 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633)
I20250110 03:07:47.758884 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
I20250110 03:07:47.758958 1619 raft_consensus.cc:3049] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:47.758930 1750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:47.758984 1619 raft_consensus.cc:738] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:47.758999 1750 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.759032 1619 consensus_queue.cc:260] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.759127 1619 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.759685 1750 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 2.
I20250110 03:07:47.759779 1619 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 2.
I20250110 03:07:47.759925 1818 leader_election.cc:304] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, 90d44e46d37f48258cea315fd7e589d5, baa5f3e8148b4fdda964dbc1cb4a30c0; no voters:
I20250110 03:07:47.760041 3085 raft_consensus.cc:2798] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:47.760162 3085 raft_consensus.cc:695] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Becoming Leader. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Running, Role: LEADER
I20250110 03:07:47.760221 3085 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.760777 1470 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: term changed from 1 to 2, leader changed from baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) to 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131). New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:47.762470 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eefff01740a34c40ba687c7242b87770" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c2867695ca20469dba7d61ebc5ecebfa"
I20250110 03:07:47.762606 2742 raft_consensus.cc:3054] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.763182 2742 raft_consensus.cc:2463] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 2.
I20250110 03:07:47.795202 3318 raft_consensus.cc:988] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: : Instructing follower 84c4433aa11f4e2f9a49bdb01aba84e0 to start an election
I20250110 03:07:47.795274 3318 raft_consensus.cc:1076] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Signalling peer 84c4433aa11f4e2f9a49bdb01aba84e0 to start an election
I20250110 03:07:47.795459 1882 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db"
dest_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0"
from {username='slave'} at 127.14.19.129:46789
I20250110 03:07:47.795527 1882 raft_consensus.cc:491] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:07:47.795547 1882 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.796042 1882 raft_consensus.cc:513] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.796191 1882 leader_election.cc:290] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 2 election: Requested vote from peers 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231), baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179), c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633)
I20250110 03:07:47.796339 1750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "90d44e46d37f48258cea315fd7e589d5"
I20250110 03:07:47.796355 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c2867695ca20469dba7d61ebc5ecebfa"
I20250110 03:07:47.796420 2742 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.796420 1750 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.796408 1619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0eee4c5c9b54dfca9b59276787773db" candidate_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0"
I20250110 03:07:47.796492 1619 raft_consensus.cc:3049] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:07:47.796530 1619 raft_consensus.cc:738] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Running, Role: LEADER
I20250110 03:07:47.796590 1619 consensus_queue.cc:260] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.796710 1619 raft_consensus.cc:3054] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:07:47.797065 1750 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 2.
I20250110 03:07:47.797065 2742 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 2.
I20250110 03:07:47.797271 1817 leader_election.cc:304] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 84c4433aa11f4e2f9a49bdb01aba84e0, 90d44e46d37f48258cea315fd7e589d5, c2867695ca20469dba7d61ebc5ecebfa; no voters:
I20250110 03:07:47.797281 1619 raft_consensus.cc:2463] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c4433aa11f4e2f9a49bdb01aba84e0 in term 2.
I20250110 03:07:47.797366 3085 raft_consensus.cc:2798] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:07:47.797422 3085 raft_consensus.cc:695] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Becoming Leader. State: Replica: 84c4433aa11f4e2f9a49bdb01aba84e0, State: Running, Role: LEADER
I20250110 03:07:47.797480 3085 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:47.797956 1470 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: term changed from 1 to 2, leader changed from baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) to 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131). New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:48.213611 1750 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250110 03:07:48.213891 3085 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:48.257923 2742 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250110 03:07:48.258172 3085 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:48.266949 2742 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250110 03:07:48.267131 3085 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:48.268055 1619 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250110 03:07:48.268352 3085 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:48.268595 1882 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:48.269039 2742 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250110 03:07:48.269191 1750 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250110 03:07:48.269228 3085 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.269361 3085 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.269711 3666 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.269825 2742 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.270016 1750 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.270642 1461 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet eefff01740a34c40ba687c7242b87770 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:48.271023 1472 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa reported cstate change: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.271890 1599 tablet_service.cc:1514] Processing DeleteTablet for tablet eefff01740a34c40ba687c7242b87770 with delete_type TABLET_DATA_TOMBSTONED (TS baa5f3e8148b4fdda964dbc1cb4a30c0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49710
I20250110 03:07:48.272075 3670 tablet_replica.cc:331] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: stopping tablet replica
I20250110 03:07:48.272159 3670 raft_consensus.cc:2238] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:48.272207 3670 raft_consensus.cc:2267] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:48.272446 3670 ts_tablet_manager.cc:1905] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:48.273567 3670 ts_tablet_manager.cc:1918] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20250110 03:07:48.273612 3670 log.cc:1198] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/eefff01740a34c40ba687c7242b87770
I20250110 03:07:48.273905 1463 catalog_manager.cc:4872] TS baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179): tablet eefff01740a34c40ba687c7242b87770 (table rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]) successfully deleted
I20250110 03:07:48.358505 1750 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250110 03:07:48.358745 3666 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:48.359826 1619 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250110 03:07:48.360311 3643 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:48.360513 1882 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:48.360961 2742 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250110 03:07:48.360968 1750 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250110 03:07:48.361168 3643 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.361277 3666 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.361740 3643 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.361900 2742 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.361977 1750 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.362784 1461 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d0eee4c5c9b54dfca9b59276787773db with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:48.363071 1473 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa reported cstate change: config changed from index 5 to 7, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:48.363797 1599 tablet_service.cc:1514] Processing DeleteTablet for tablet d0eee4c5c9b54dfca9b59276787773db with delete_type TABLET_DATA_TOMBSTONED (TS baa5f3e8148b4fdda964dbc1cb4a30c0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49710
I20250110 03:07:48.363929 3670 tablet_replica.cc:331] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: stopping tablet replica
I20250110 03:07:48.363999 3670 raft_consensus.cc:2238] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:48.364048 3670 raft_consensus.cc:2267] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:48.364351 3670 ts_tablet_manager.cc:1905] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:48.365528 3670 ts_tablet_manager.cc:1918] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20250110 03:07:48.365582 3670 log.cc:1198] T d0eee4c5c9b54dfca9b59276787773db P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/d0eee4c5c9b54dfca9b59276787773db
I20250110 03:07:48.365883 1463 catalog_manager.cc:4872] TS baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179): tablet d0eee4c5c9b54dfca9b59276787773db (table rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]) successfully deleted
I20250110 03:07:48.548213 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:48.548682 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:48.548883 1863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:48.552933 1599 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:48.553921 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:48.561226 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:48.594609 1863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:48.595270 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:48.596238 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:48.596765 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:48.597416 1599 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:48.606482 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:48.616662 1619 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } }
I20250110 03:07:48.617182 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:48.617252 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:48.617369 3397 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.617466 3318 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:48.617563 1553 consensus_peers.cc:487] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): Couldn't send request to peer 90d44e46d37f48258cea315fd7e589d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.618229 3318 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } }
I20250110 03:07:48.618353 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } }
I20250110 03:07:48.618371 1882 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } }
I20250110 03:07:48.619448 1472 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:48.620173 1619 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
W20250110 03:07:48.620663 1553 consensus_peers.cc:487] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): Couldn't send request to peer 90d44e46d37f48258cea315fd7e589d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.620684 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:48.620738 1552 consensus_peers.cc:487] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer 856267e243cb46ab884b7a1694ae465e (127.14.19.132:40673): Couldn't send request to peer 856267e243cb46ab884b7a1694ae465e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d68741aa20be48229a19dda143743bca. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.620875 3397 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:48.620956 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:48.621214 3397 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:48.621660 3787 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:48.621868 1882 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:48.622017 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:48.623028 1470 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 7 to 8, NON_VOTER 856267e243cb46ab884b7a1694ae465e (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } } }
I20250110 03:07:48.625715 1882 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } }
I20250110 03:07:48.626164 2611 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20250110 03:07:48.626282 1817 consensus_peers.cc:487] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 -> Peer 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): Couldn't send request to peer 90d44e46d37f48258cea315fd7e589d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.626343 3666 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.626724 1619 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:48.626925 3666 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.627092 3687 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } }
I20250110 03:07:48.627209 2611 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } }
I20250110 03:07:48.628029 1619 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } }
I20250110 03:07:48.628188 1470 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 6 to 7, NON_VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) added. New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:48.629458 1882 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:48.629932 2742 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 03:07:48.629930 1750 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20250110 03:07:48.630112 1819 consensus_peers.cc:487] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 -> Peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075): Couldn't send request to peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0eee4c5c9b54dfca9b59276787773db. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.630124 3666 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:48.630221 3643 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:48.630843 3085 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:48.630959 1750 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:48.630964 2742 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:48.632254 1473 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: config changed from index 7 to 8, NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) added. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } } }
I20250110 03:07:48.634253 1882 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } }
I20250110 03:07:48.634670 2742 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 03:07:48.634670 1750 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20250110 03:07:48.634824 1818 consensus_peers.cc:487] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 -> Peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179): Couldn't send request to peer baa5f3e8148b4fdda964dbc1cb4a30c0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.634855 3666 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:48.634943 3643 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:48.635782 3666 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } } }
I20250110 03:07:48.635910 2742 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } } }
I20250110 03:07:48.636904 1470 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa reported cstate change: config changed from index 7 to 8, NON_VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) added. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } } }
I20250110 03:07:48.636947 1750 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } } }
I20250110 03:07:48.638125 1619 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:48.638617 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:48.638653 2611 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:48.638808 3787 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:48.638912 3397 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:48.639098 1555 consensus_peers.cc:487] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 -> Peer c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633): Couldn't send request to peer c2867695ca20469dba7d61ebc5ecebfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56aea05a8a714b4ca357e24049c7f348. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:48.639559 3318 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:48.639693 1882 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:48.640739 1472 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 6 to 7, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:48.641016 2611 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } } }
I20250110 03:07:49.007779 3873 ts_tablet_manager.cc:927] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:49.008035 3873 tablet_copy_client.cc:323] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:49.008431 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet d68741aa20be48229a19dda143743bca from peer 90d44e46d37f48258cea315fd7e589d5 ({username='slave'} at 127.14.19.130:44659)
I20250110 03:07:49.008495 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet d68741aa20be48229a19dda143743bca from peer 90d44e46d37f48258cea315fd7e589d5 at {username='slave'} at 127.14.19.130:44659: session id = 90d44e46d37f48258cea315fd7e589d5-d68741aa20be48229a19dda143743bca
I20250110 03:07:49.008872 1639 tablet_copy_source_session.cc:215] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:49.009253 3873 ts_tablet_manager.cc:1905] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:49.010802 3873 ts_tablet_manager.cc:1918] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250110 03:07:49.010926 3873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d68741aa20be48229a19dda143743bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:49.011544 3873 tablet_copy_client.cc:806] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:49.011680 3873 tablet_copy_client.cc:670] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:49.012374 3873 tablet_copy_client.cc:538] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:49.013044 3873 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:49.015791 3873 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:49.015966 3873 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Bootstrap complete.
I20250110 03:07:49.016044 3873 ts_tablet_manager.cc:1397] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:49.016150 3873 raft_consensus.cc:357] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:49.016214 3873 raft_consensus.cc:738] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: LEARNER
I20250110 03:07:49.016237 3873 consensus_queue.cc:260] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:49.016337 3873 ts_tablet_manager.cc:1428] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:49.016571 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session 90d44e46d37f48258cea315fd7e589d5-d68741aa20be48229a19dda143743bca received from {username='slave'} at 127.14.19.130:44659
I20250110 03:07:49.016628 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session 90d44e46d37f48258cea315fd7e589d5-d68741aa20be48229a19dda143743bca on tablet d68741aa20be48229a19dda143743bca with peer 90d44e46d37f48258cea315fd7e589d5
I20250110 03:07:49.049110 3876 ts_tablet_manager.cc:927] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:49.049299 3876 tablet_copy_client.cc:323] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:49.049552 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet 56aea05a8a714b4ca357e24049c7f348 from peer c2867695ca20469dba7d61ebc5ecebfa ({username='slave'} at 127.14.19.133:45019)
I20250110 03:07:49.049614 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet 56aea05a8a714b4ca357e24049c7f348 from peer c2867695ca20469dba7d61ebc5ecebfa at {username='slave'} at 127.14.19.133:45019: session id = c2867695ca20469dba7d61ebc5ecebfa-56aea05a8a714b4ca357e24049c7f348
I20250110 03:07:49.050007 1639 tablet_copy_source_session.cc:215] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:49.050396 3876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aea05a8a714b4ca357e24049c7f348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:49.051350 3876 tablet_copy_client.cc:806] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:49.051456 3876 tablet_copy_client.cc:670] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:49.052093 3876 tablet_copy_client.cc:538] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:49.052603 3876 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap starting.
I20250110 03:07:49.052997 3878 ts_tablet_manager.cc:927] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: Initiating tablet copy from peer baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179)
I20250110 03:07:49.053143 3878 tablet_copy_client.cc:323] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:33179
I20250110 03:07:49.053341 1639 tablet_copy_service.cc:140] P baa5f3e8148b4fdda964dbc1cb4a30c0: Received BeginTabletCopySession request for tablet d68741aa20be48229a19dda143743bca from peer 856267e243cb46ab884b7a1694ae465e ({username='slave'} at 127.14.19.132:37257)
I20250110 03:07:49.053406 1639 tablet_copy_service.cc:161] P baa5f3e8148b4fdda964dbc1cb4a30c0: Beginning new tablet copy session on tablet d68741aa20be48229a19dda143743bca from peer 856267e243cb46ab884b7a1694ae465e at {username='slave'} at 127.14.19.132:37257: session id = 856267e243cb46ab884b7a1694ae465e-d68741aa20be48229a19dda143743bca
I20250110 03:07:49.053884 1639 tablet_copy_source_session.cc:215] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:49.054211 3878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d68741aa20be48229a19dda143743bca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:49.055155 3878 tablet_copy_client.cc:806] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:49.055266 3878 tablet_copy_client.cc:670] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:49.055811 3878 tablet_copy_client.cc:538] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:49.056310 3878 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: Bootstrap starting.
I20250110 03:07:49.056373 3876 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:49.056558 3876 tablet_bootstrap.cc:492] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: Bootstrap complete.
I20250110 03:07:49.056648 3876 ts_tablet_manager.cc:1397] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250110 03:07:49.056762 3876 raft_consensus.cc:357] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:49.056816 3876 raft_consensus.cc:738] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2867695ca20469dba7d61ebc5ecebfa, State: Initialized, Role: LEARNER
I20250110 03:07:49.056862 3876 consensus_queue.cc:260] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: true } }
I20250110 03:07:49.056952 3876 ts_tablet_manager.cc:1428] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:49.057161 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session c2867695ca20469dba7d61ebc5ecebfa-56aea05a8a714b4ca357e24049c7f348 received from {username='slave'} at 127.14.19.133:45019
I20250110 03:07:49.057219 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session c2867695ca20469dba7d61ebc5ecebfa-56aea05a8a714b4ca357e24049c7f348 on tablet 56aea05a8a714b4ca357e24049c7f348 with peer c2867695ca20469dba7d61ebc5ecebfa
I20250110 03:07:49.059522 3878 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:49.059724 3878 tablet_bootstrap.cc:492] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: Bootstrap complete.
I20250110 03:07:49.059814 3878 ts_tablet_manager.cc:1397] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250110 03:07:49.059911 3878 raft_consensus.cc:357] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:49.059970 3878 raft_consensus.cc:738] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 856267e243cb46ab884b7a1694ae465e, State: Initialized, Role: LEARNER
I20250110 03:07:49.060005 3878 consensus_queue.cc:260] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: true } }
I20250110 03:07:49.060165 3878 ts_tablet_manager.cc:1428] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:07:49.060359 1639 tablet_copy_service.cc:342] P baa5f3e8148b4fdda964dbc1cb4a30c0: Request end of tablet copy session 856267e243cb46ab884b7a1694ae465e-d68741aa20be48229a19dda143743bca received from {username='slave'} at 127.14.19.132:37257
I20250110 03:07:49.060411 1639 tablet_copy_service.cc:434] P baa5f3e8148b4fdda964dbc1cb4a30c0: ending tablet copy session 856267e243cb46ab884b7a1694ae465e-d68741aa20be48229a19dda143743bca on tablet d68741aa20be48229a19dda143743bca with peer 856267e243cb46ab884b7a1694ae465e
I20250110 03:07:49.087368 3873 ts_tablet_manager.cc:927] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Initiating tablet copy from peer 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:49.087508 3873 tablet_copy_client.cc:323] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:42095
I20250110 03:07:49.087707 1903 tablet_copy_service.cc:140] P 84c4433aa11f4e2f9a49bdb01aba84e0: Received BeginTabletCopySession request for tablet 48cdbe84b6094cccaf69b553fd8f1d68 from peer 90d44e46d37f48258cea315fd7e589d5 ({username='slave'} at 127.14.19.130:59253)
I20250110 03:07:49.087764 1903 tablet_copy_service.cc:161] P 84c4433aa11f4e2f9a49bdb01aba84e0: Beginning new tablet copy session on tablet 48cdbe84b6094cccaf69b553fd8f1d68 from peer 90d44e46d37f48258cea315fd7e589d5 at {username='slave'} at 127.14.19.130:59253: session id = 90d44e46d37f48258cea315fd7e589d5-48cdbe84b6094cccaf69b553fd8f1d68
I20250110 03:07:49.088124 1903 tablet_copy_source_session.cc:215] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:49.088380 3873 ts_tablet_manager.cc:1905] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:49.089699 3873 ts_tablet_manager.cc:1918] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250110 03:07:49.089784 3873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cdbe84b6094cccaf69b553fd8f1d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:49.090288 3873 tablet_copy_client.cc:806] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:49.090374 3873 tablet_copy_client.cc:670] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:49.090942 3873 tablet_copy_client.cc:538] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:49.091421 3873 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap starting.
I20250110 03:07:49.093755 3873 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:49.093909 3873 tablet_bootstrap.cc:492] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Bootstrap complete.
I20250110 03:07:49.093974 3873 ts_tablet_manager.cc:1397] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:07:49.094054 3873 raft_consensus.cc:357] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } }
I20250110 03:07:49.094108 3873 raft_consensus.cc:738] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90d44e46d37f48258cea315fd7e589d5, State: Initialized, Role: LEARNER
I20250110 03:07:49.094128 3873 consensus_queue.cc:260] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } }
I20250110 03:07:49.094234 3873 ts_tablet_manager.cc:1428] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:49.094425 1903 tablet_copy_service.cc:342] P 84c4433aa11f4e2f9a49bdb01aba84e0: Request end of tablet copy session 90d44e46d37f48258cea315fd7e589d5-48cdbe84b6094cccaf69b553fd8f1d68 received from {username='slave'} at 127.14.19.130:59253
I20250110 03:07:49.094480 1903 tablet_copy_service.cc:434] P 84c4433aa11f4e2f9a49bdb01aba84e0: ending tablet copy session 90d44e46d37f48258cea315fd7e589d5-48cdbe84b6094cccaf69b553fd8f1d68 on tablet 48cdbe84b6094cccaf69b553fd8f1d68 with peer 90d44e46d37f48258cea315fd7e589d5
I20250110 03:07:49.106639 3915 ts_tablet_manager.cc:927] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Initiating tablet copy from peer 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:49.106798 3915 tablet_copy_client.cc:323] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:42095
I20250110 03:07:49.107072 1903 tablet_copy_service.cc:140] P 84c4433aa11f4e2f9a49bdb01aba84e0: Received BeginTabletCopySession request for tablet eefff01740a34c40ba687c7242b87770 from peer baa5f3e8148b4fdda964dbc1cb4a30c0 ({username='slave'} at 127.14.19.129:46789)
I20250110 03:07:49.107131 1903 tablet_copy_service.cc:161] P 84c4433aa11f4e2f9a49bdb01aba84e0: Beginning new tablet copy session on tablet eefff01740a34c40ba687c7242b87770 from peer baa5f3e8148b4fdda964dbc1cb4a30c0 at {username='slave'} at 127.14.19.129:46789: session id = baa5f3e8148b4fdda964dbc1cb4a30c0-eefff01740a34c40ba687c7242b87770
I20250110 03:07:49.107551 1903 tablet_copy_source_session.cc:215] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:49.107856 3915 ts_tablet_manager.cc:1905] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:07:49.109160 3915 ts_tablet_manager.cc:1918] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20250110 03:07:49.109256 3915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eefff01740a34c40ba687c7242b87770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:49.109757 3915 tablet_copy_client.cc:806] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:49.109863 3915 tablet_copy_client.cc:670] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:49.110513 3915 tablet_copy_client.cc:538] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:49.111039 3915 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap starting.
I20250110 03:07:49.114009 3915 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:49.114218 3915 tablet_bootstrap.cc:492] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Bootstrap complete.
I20250110 03:07:49.114306 3915 ts_tablet_manager.cc:1397] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:49.114405 3915 raft_consensus.cc:357] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } }
I20250110 03:07:49.114464 3915 raft_consensus.cc:738] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: baa5f3e8148b4fdda964dbc1cb4a30c0, State: Initialized, Role: LEARNER
I20250110 03:07:49.114496 3915 consensus_queue.cc:260] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: NON_VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: true } }
I20250110 03:07:49.114614 3915 ts_tablet_manager.cc:1428] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:07:49.114846 1903 tablet_copy_service.cc:342] P 84c4433aa11f4e2f9a49bdb01aba84e0: Request end of tablet copy session baa5f3e8148b4fdda964dbc1cb4a30c0-eefff01740a34c40ba687c7242b87770 received from {username='slave'} at 127.14.19.129:46789
I20250110 03:07:49.114897 1903 tablet_copy_service.cc:434] P 84c4433aa11f4e2f9a49bdb01aba84e0: ending tablet copy session baa5f3e8148b4fdda964dbc1cb4a30c0-eefff01740a34c40ba687c7242b87770 on tablet eefff01740a34c40ba687c7242b87770 with peer baa5f3e8148b4fdda964dbc1cb4a30c0
I20250110 03:07:49.158520 3918 ts_tablet_manager.cc:927] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Initiating tablet copy from peer 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095)
I20250110 03:07:49.158723 3918 tablet_copy_client.cc:323] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:42095
I20250110 03:07:49.158954 1903 tablet_copy_service.cc:140] P 84c4433aa11f4e2f9a49bdb01aba84e0: Received BeginTabletCopySession request for tablet d0eee4c5c9b54dfca9b59276787773db from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec ({username='slave'} at 127.14.19.134:43625)
I20250110 03:07:49.159016 1903 tablet_copy_service.cc:161] P 84c4433aa11f4e2f9a49bdb01aba84e0: Beginning new tablet copy session on tablet d0eee4c5c9b54dfca9b59276787773db from peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec at {username='slave'} at 127.14.19.134:43625: session id = 9ddba97e3a4f4b4b99e67a34c4a0e3ec-d0eee4c5c9b54dfca9b59276787773db
I20250110 03:07:49.159392 1903 tablet_copy_source_session.cc:215] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:49.159694 3918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0eee4c5c9b54dfca9b59276787773db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:49.160596 3918 tablet_copy_client.cc:806] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:49.160727 3918 tablet_copy_client.cc:670] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:49.161293 3918 tablet_copy_client.cc:538] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:49.161769 3918 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap starting.
I20250110 03:07:49.164762 3918 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:49.164958 3918 tablet_bootstrap.cc:492] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Bootstrap complete.
I20250110 03:07:49.165031 3918 ts_tablet_manager.cc:1397] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:49.165117 3918 raft_consensus.cc:357] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:49.165184 3918 raft_consensus.cc:738] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9ddba97e3a4f4b4b99e67a34c4a0e3ec, State: Initialized, Role: LEARNER
I20250110 03:07:49.165205 3918 consensus_queue.cc:260] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: NON_VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: true } }
I20250110 03:07:49.165320 3918 ts_tablet_manager.cc:1428] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Time spent starting tablet: real 0.000s user 0.001s sys 0.001s
I20250110 03:07:49.165577 1903 tablet_copy_service.cc:342] P 84c4433aa11f4e2f9a49bdb01aba84e0: Request end of tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-d0eee4c5c9b54dfca9b59276787773db received from {username='slave'} at 127.14.19.134:43625
I20250110 03:07:49.165632 1903 tablet_copy_service.cc:434] P 84c4433aa11f4e2f9a49bdb01aba84e0: ending tablet copy session 9ddba97e3a4f4b4b99e67a34c4a0e3ec-d0eee4c5c9b54dfca9b59276787773db on tablet d0eee4c5c9b54dfca9b59276787773db with peer 9ddba97e3a4f4b4b99e67a34c4a0e3ec
I20250110 03:07:49.411080 1750 raft_consensus.cc:1212] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:49.499078 2873 raft_consensus.cc:1212] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:07:49.513701 2742 raft_consensus.cc:1212] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:49.568926 1750 raft_consensus.cc:1212] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20250110 03:07:49.577020 1618 raft_consensus.cc:1212] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250110 03:07:49.737388 2611 raft_consensus.cc:1212] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250110 03:07:49.946447 3963 raft_consensus.cc:1059] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER c2867695ca20469dba7d61ebc5ecebfa to VOTER
I20250110 03:07:49.946658 3963 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:49.947135 2742 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:49.947180 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:49.947196 2611 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:49.947336 4048 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:49.947484 3952 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:49.947597 3318 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:49.948293 3318 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.948426 2742 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.948509 2611 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.949393 1882 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.949545 1470 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 7 to 8, c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:49.950893 1618 consensus_queue.cc:237] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } }
I20250110 03:07:49.951424 1882 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:49.951462 2742 raft_consensus.cc:1270] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:49.951592 4048 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.951709 3952 consensus_queue.cc:1035] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.952275 4048 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.952425 2742 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P c2867695ca20469dba7d61ebc5ecebfa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.952431 1882 raft_consensus.cc:2949] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.953295 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 56aea05a8a714b4ca357e24049c7f348 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:49.953332 1472 catalog_manager.cc:5526] T 56aea05a8a714b4ca357e24049c7f348 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 8 to 9, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } }
I20250110 03:07:49.956115 2591 tablet_service.cc:1514] Processing DeleteTablet for tablet 56aea05a8a714b4ca357e24049c7f348 with delete_type TABLET_DATA_TOMBSTONED (TS 9ddba97e3a4f4b4b99e67a34c4a0e3ec not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50754
I20250110 03:07:49.956293 4058 tablet_replica.cc:331] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: stopping tablet replica
I20250110 03:07:49.956389 4058 raft_consensus.cc:2238] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:49.956470 4058 raft_consensus.cc:2267] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:49.956725 4058 ts_tablet_manager.cc:1905] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:49.957865 4058 ts_tablet_manager.cc:1918] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:49.957931 4058 log.cc:1198] T 56aea05a8a714b4ca357e24049c7f348 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/56aea05a8a714b4ca357e24049c7f348
I20250110 03:07:49.958261 1464 catalog_manager.cc:4872] TS 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075): tablet 56aea05a8a714b4ca357e24049c7f348 (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:49.971589 3952 raft_consensus.cc:1059] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER 856267e243cb46ab884b7a1694ae465e to VOTER
I20250110 03:07:49.971765 3952 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:49.972245 1882 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:49.972236 2873 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:49.972245 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:49.972306 1750 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:49.972448 4060 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.972543 3952 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.972608 3963 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.972703 3318 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.973479 3952 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.973587 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.973749 1750 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.973810 2873 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.974367 3952 raft_consensus.cc:1059] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0: attempting to promote NON_VOTER 90d44e46d37f48258cea315fd7e589d5 to VOTER
I20250110 03:07:49.974472 3952 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:49.975008 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:49.974973 1474 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 reported cstate change: config changed from index 8 to 9, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.975175 3952 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:49.975231 1881 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:49.975641 1882 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 856267e243cb46ab884b7a1694ae465e (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: true } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.975790 3963 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:49.976635 1750 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:49.976796 2873 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:07:49.976801 4048 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:49.976917 4048 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.976976 3318 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:49.977146 1881 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.977515 1750 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.978220 2873 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.978510 1472 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 9 to 10, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.978786 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } attrs { replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.979820 1618 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:49.980178 2873 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:49.980201 1750 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:49.980227 2611 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20250110 03:07:49.980406 3318 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:49.980474 3952 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:49.980713 3952 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:07:49.981057 3318 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.981211 2873 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.981416 1750 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.982264 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d68741aa20be48229a19dda143743bca with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:49.982455 1470 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 10 to 11, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:49.982721 2611 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false replace: true } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.983259 1618 consensus_queue.cc:237] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } }
I20250110 03:07:49.983443 1863 tablet_service.cc:1514] Processing DeleteTablet for tablet d68741aa20be48229a19dda143743bca with delete_type TABLET_DATA_TOMBSTONED (TS 84c4433aa11f4e2f9a49bdb01aba84e0 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:54160
I20250110 03:07:49.983726 1751 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:49.983732 4065 tablet_replica.cc:331] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: stopping tablet replica
I20250110 03:07:49.983796 4065 raft_consensus.cc:2238] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:49.983886 2873 raft_consensus.cc:1270] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Refusing update from remote peer baa5f3e8148b4fdda964dbc1cb4a30c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20250110 03:07:49.983913 4065 raft_consensus.cc:2267] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:49.984017 3952 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:49.984091 4060 consensus_queue.cc:1035] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250110 03:07:49.984136 4065 ts_tablet_manager.cc:1905] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:49.984615 3952 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.984872 2873 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 856267e243cb46ab884b7a1694ae465e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.985399 4065 ts_tablet_manager.cc:1918] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20250110 03:07:49.985495 4065 log.cc:1198] T d68741aa20be48229a19dda143743bca P 84c4433aa11f4e2f9a49bdb01aba84e0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/d68741aa20be48229a19dda143743bca
I20250110 03:07:49.985556 1463 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d68741aa20be48229a19dda143743bca with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:49.985714 1472 catalog_manager.cc:5526] T d68741aa20be48229a19dda143743bca P baa5f3e8148b4fdda964dbc1cb4a30c0 reported cstate change: config changed from index 11 to 12, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New cstate: current_term: 1 leader_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:49.985749 1751 raft_consensus.cc:2949] T d68741aa20be48229a19dda143743bca P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } peers { permanent_uuid: "856267e243cb46ab884b7a1694ae465e" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 40673 } attrs { promote: false } } }
I20250110 03:07:49.986013 1461 catalog_manager.cc:4872] TS 84c4433aa11f4e2f9a49bdb01aba84e0 (127.14.19.131:42095): tablet d68741aa20be48229a19dda143743bca (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:49.986474 2591 tablet_service.cc:1514] Processing DeleteTablet for tablet d68741aa20be48229a19dda143743bca with delete_type TABLET_DATA_TOMBSTONED (TS 9ddba97e3a4f4b4b99e67a34c4a0e3ec not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:50754
I20250110 03:07:49.986590 4058 tablet_replica.cc:331] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: stopping tablet replica
I20250110 03:07:49.986656 4058 raft_consensus.cc:2238] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:49.986729 4058 raft_consensus.cc:2267] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:49.986945 4058 ts_tablet_manager.cc:1905] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:49.987995 4058 ts_tablet_manager.cc:1918] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20250110 03:07:49.988044 4058 log.cc:1198] T d68741aa20be48229a19dda143743bca P 9ddba97e3a4f4b4b99e67a34c4a0e3ec: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-5/wal/wals/d68741aa20be48229a19dda143743bca
I20250110 03:07:49.988340 1464 catalog_manager.cc:4872] TS 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134:38075): tablet d68741aa20be48229a19dda143743bca (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:50.001621 3643 raft_consensus.cc:1059] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0: attempting to promote NON_VOTER 90d44e46d37f48258cea315fd7e589d5 to VOTER
I20250110 03:07:50.001768 3643 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } }
I20250110 03:07:50.002180 1751 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 LEARNER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:50.002264 2611 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:50.002295 1618 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:50.002384 4067 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:50.002511 3984 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:50.002628 3643 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:50.003204 3643 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.003530 1618 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.004019 1751 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.004158 1470 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 7 to 8, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:50.004882 2611 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.006028 1881 consensus_queue.cc:237] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } }
I20250110 03:07:50.006381 2611 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:50.006429 1751 raft_consensus.cc:1270] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:07:50.006594 3985 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.006695 4067 consensus_queue.cc:1035] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.007249 3984 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.007360 2611 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.007917 1461 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 48cdbe84b6094cccaf69b553fd8f1d68 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:50.008051 1751 raft_consensus.cc:2949] T 48cdbe84b6094cccaf69b553fd8f1d68 P 90d44e46d37f48258cea315fd7e589d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } } }
I20250110 03:07:50.008090 1473 catalog_manager.cc:5526] T 48cdbe84b6094cccaf69b553fd8f1d68 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 8 to 9, VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:50.008739 1599 tablet_service.cc:1514] Processing DeleteTablet for tablet 48cdbe84b6094cccaf69b553fd8f1d68 with delete_type TABLET_DATA_TOMBSTONED (TS baa5f3e8148b4fdda964dbc1cb4a30c0 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49710
I20250110 03:07:50.009089 4071 tablet_replica.cc:331] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: stopping tablet replica
I20250110 03:07:50.009153 4071 raft_consensus.cc:2238] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:50.009241 4071 raft_consensus.cc:2267] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:50.009433 4071 ts_tablet_manager.cc:1905] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:50.010476 4071 ts_tablet_manager.cc:1918] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20250110 03:07:50.010521 4071 log.cc:1198] T 48cdbe84b6094cccaf69b553fd8f1d68 P baa5f3e8148b4fdda964dbc1cb4a30c0: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/48cdbe84b6094cccaf69b553fd8f1d68
I20250110 03:07:50.012219 1463 catalog_manager.cc:4872] TS baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129:33179): tablet 48cdbe84b6094cccaf69b553fd8f1d68 (table rebalance_test_table_1 [id=924c92d998df4505a2cf1395cf34abc3]) successfully deleted
I20250110 03:07:50.145576 4067 raft_consensus.cc:1059] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0: attempting to promote NON_VOTER baa5f3e8148b4fdda964dbc1cb4a30c0 to VOTER
I20250110 03:07:50.145759 4067 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } }
I20250110 03:07:50.146240 1751 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:50.146390 2742 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:50.146595 3985 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.146682 1618 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 LEARNER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:50.147008 3985 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.147125 3643 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.147713 3643 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.148712 1751 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.148838 1618 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.149606 2742 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.149845 1473 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 8 to 9, baa5f3e8148b4fdda964dbc1cb4a30c0 (127.14.19.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:50.151742 1881 consensus_queue.cc:237] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } }
I20250110 03:07:50.152158 2742 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:50.152158 1618 raft_consensus.cc:1270] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:50.152328 3643 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:50.152418 3985 consensus_queue.cc:1035] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:50.152931 3985 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.153066 1618 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P baa5f3e8148b4fdda964dbc1cb4a30c0 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.153051 2742 raft_consensus.cc:2949] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.153685 1461 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet eefff01740a34c40ba687c7242b87770 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:50.154057 1473 catalog_manager.cc:5526] T eefff01740a34c40ba687c7242b87770 P c2867695ca20469dba7d61ebc5ecebfa reported cstate change: config changed from index 9 to 10, VOTER 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130) evicted. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false } } peers { permanent_uuid: "baa5f3e8148b4fdda964dbc1cb4a30c0" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 33179 } attrs { promote: false } } }
I20250110 03:07:50.154760 1729 tablet_service.cc:1514] Processing DeleteTablet for tablet eefff01740a34c40ba687c7242b87770 with delete_type TABLET_DATA_TOMBSTONED (TS 90d44e46d37f48258cea315fd7e589d5 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56738
I20250110 03:07:50.154938 4116 tablet_replica.cc:331] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: stopping tablet replica
I20250110 03:07:50.155006 4116 raft_consensus.cc:2238] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:50.155103 4116 raft_consensus.cc:2267] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:50.155344 4116 ts_tablet_manager.cc:1905] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:50.156441 4116 ts_tablet_manager.cc:1918] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20250110 03:07:50.156486 4116 log.cc:1198] T eefff01740a34c40ba687c7242b87770 P 90d44e46d37f48258cea315fd7e589d5: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/eefff01740a34c40ba687c7242b87770
I20250110 03:07:50.156771 1462 catalog_manager.cc:4872] TS 90d44e46d37f48258cea315fd7e589d5 (127.14.19.130:38231): tablet eefff01740a34c40ba687c7242b87770 (table rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]) successfully deleted
I20250110 03:07:50.253329 4067 raft_consensus.cc:1059] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0: attempting to promote NON_VOTER 9ddba97e3a4f4b4b99e67a34c4a0e3ec to VOTER
I20250110 03:07:50.253494 4067 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:50.253984 2611 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 LEARNER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:50.254014 1751 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:50.254033 2742 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250110 03:07:50.254231 3985 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.254366 3984 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.254475 4067 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:07:50.255132 3985 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.255246 1751 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.256023 2742 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.256110 2611 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.256160 1474 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 reported cstate change: config changed from index 8 to 9, 9ddba97e3a4f4b4b99e67a34c4a0e3ec (127.14.19.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2867695ca20469dba7d61ebc5ecebfa" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 36633 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:50.257175 1881 consensus_queue.cc:237] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } }
I20250110 03:07:50.257582 1751 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:50.257620 2611 raft_consensus.cc:1270] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 FOLLOWER]: Refusing update from remote peer 84c4433aa11f4e2f9a49bdb01aba84e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:07:50.257834 3984 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:50.257920 3643 consensus_queue.cc:1035] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:07:50.258450 3643 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 84c4433aa11f4e2f9a49bdb01aba84e0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.258641 2611 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.258667 1751 raft_consensus.cc:2949] T d0eee4c5c9b54dfca9b59276787773db P 90d44e46d37f48258cea315fd7e589d5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.259440 1461 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d0eee4c5c9b54dfca9b59276787773db with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:50.259476 1474 catalog_manager.cc:5526] T d0eee4c5c9b54dfca9b59276787773db P 9ddba97e3a4f4b4b99e67a34c4a0e3ec reported cstate change: config changed from index 9 to 10, VOTER c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133) evicted. New cstate: current_term: 2 leader_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "90d44e46d37f48258cea315fd7e589d5" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 38231 } } peers { permanent_uuid: "84c4433aa11f4e2f9a49bdb01aba84e0" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 42095 } } peers { permanent_uuid: "9ddba97e3a4f4b4b99e67a34c4a0e3ec" member_type: VOTER last_known_addr { host: "127.14.19.134" port: 38075 } attrs { promote: false } } }
I20250110 03:07:50.262615 2722 tablet_service.cc:1514] Processing DeleteTablet for tablet d0eee4c5c9b54dfca9b59276787773db with delete_type TABLET_DATA_TOMBSTONED (TS c2867695ca20469dba7d61ebc5ecebfa not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49010
I20250110 03:07:50.262820 4122 tablet_replica.cc:331] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: stopping tablet replica
I20250110 03:07:50.262914 4122 raft_consensus.cc:2238] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:50.262998 4122 raft_consensus.cc:2267] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:50.263254 4122 ts_tablet_manager.cc:1905] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:50.264441 4122 ts_tablet_manager.cc:1918] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20250110 03:07:50.264496 4122 log.cc:1198] T d0eee4c5c9b54dfca9b59276787773db P c2867695ca20469dba7d61ebc5ecebfa: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/d0eee4c5c9b54dfca9b59276787773db
I20250110 03:07:50.264822 1464 catalog_manager.cc:4872] TS c2867695ca20469dba7d61ebc5ecebfa (127.14.19.133:36633): tablet d0eee4c5c9b54dfca9b59276787773db (table rebalance_test_table_2 [id=16527c7f48bd4f6b9d141bb2928a5aa6]) successfully deleted
I20250110 03:07:50.380772 1863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:50.384510 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:50.388622 1599 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:50.389672 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:50.390030 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:50.390673 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:50.424244 2591 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:50.425503 1729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:50.428328 1599 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:50.429473 2722 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:50.429819 1863 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20250110 03:07:50.434346 2853 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:2130: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskWQrDfu/build/release/bin/kudu: process exited on signal 11 (core dumped)
stdout: Locations load summary:
Location | Load
----------+----------
/A | 9.000000
/B | 0.000000
--------------------------------------------------
Location: /A
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+----------
Minimum Replica Count | 9
Maximum Replica Count | 9
Average Replica Count | 9.000000
Per-server replica distribution details:
UUID | Address | Replica Count
----------------------------------+---------------------+---------------
84c4433aa11f4e2f9a49bdb01aba84e0 | 127.14.19.131:42095 | 9
90d44e46d37f48258cea315fd7e589d5 | 127.14.19.130:38231 | 9
baa5f3e8148b4fdda964dbc1cb4a30c0 | 127.14.19.129:33179 | 9
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 0
Maximum | 0
Average | 0.000000
Per-table replica distribution details for tables:
Table Id | Replica Count | Replica Skew | Table Name
----------------------------------+---------------+--------------+------------------------
b8710a737ffd4f5f93b9e9df5862bb05 | 9 | 0 | rebalance_test_table_0
924c92d998df4505a2cf1395cf34abc3 | 9 | 0 | rebalance_test_table_1
16527c7f48bd4f6b9d141bb2928a5aa6 | 9 | 0 | rebalance_test_table_2
--------------------------------------------------
Location: /B
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+----------
Minimum Replica Count | 0
Maximum Replica Count | 0
Average Replica Count | 0.000000
Per-server replica distribution details:
UUID | Address | Replica Count
----------------------------------+---------------------+---------------
856267e243cb46ab884b7a1694ae465e | 127.14.19.132:40673 | 0
c2867695ca20469dba7d61ebc5ecebfa | 127.14.19.133:36633 | 0
9ddba97e3a4f4b4b99e67a34c4a0e3ec | 127.14.19.134:38075 | 0
Per-table replica distribution summary:
Replica Skew | Value
--------------+-------
N/A | N/A
Per-table replica distribution details for tables:
Table Id | Replica Count | Replica Skew | Table Name
----------+---------------+--------------+------------
Placement policy violations:
Location | Number of non-complying tables | Number of non-complying tablets
----------+--------------------------------+---------------------------------
/A | 3 | 9
Placement policy violation details:
Location | Table Name | Tablet | RF | Replicas at location
----------+------------------------+----------------------------------+----+----------------------
/A | rebalance_test_table_0 | b373739a50754ecb8753a84d813c0151 | 3 | 3
/A | rebalance_test_table_0 | 0a02da6c3e2d4db5b1db2f1de6824bc5 | 3 | 3
/A | rebalance_test_table_0 | 6fde79de661a4993ace77eadfc877bf4 | 3 | 3
/A | rebalance_test_table_1 | 48cdbe84b6094cccaf69b553fd8f1d68 | 3 | 3
/A | rebalance_test_table_2 | eefff01740a34c40ba687c7242b87770 | 3 | 3
/A | rebalance_test_table_1 | 56aea05a8a714b4ca357e24049c7f348 | 3 | 3
/A | rebalance_test_table_2 | a4054e1b6e654bc4b57d42be2a85296e | 3 | 3
/A | rebalance_test_table_1 | d68741aa20be48229a19dda143743bca | 3 | 3
/A | rebalance_test_table_2 | d0eee4c5c9b54dfca9b59276787773db | 3 | 3
rebalancing is complete: cluster is balanced (moved 14 replicas)
Locations load summary:
Location | Load
----------+----------
/A | 6.333333
/B | 2.666667
--------------------------------------------------
Location: /A
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+----------
Minimum Replica Count | 6
Maximum Replica Count | 7
Average Replica Count | 6.333333
Per-server replica distribution details:
UUID | Address | Replica Count
----------------------------------+---------------------+---------------
84c4433aa11f4e2f9a49bdb01aba84e0 | 127.14.19.131:42095 | 6
baa5f3e8148b4fdda964dbc1cb4a30c0 | 127.14.19.129:33179 | 6
90d44e46d37f48258cea315fd7e589d5 | 127.14.19.130:38231 | 7
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 0
Maximum | 1
Average | 0.333333
Per-table replica distribution details for tables:
Table Id | Replica Count | Replica Skew | Table Name
----------------------------------+---------------+--------------+------------------------
16527c7f48bd4f6b9d141bb2928a5aa6 | 9 | 0 | rebalance_test_table_2
924c92d998df4505a2cf1395cf34abc3 | 9 | 0 | rebalance_test_table_1
b8710a737ffd4f5f93b9e9df5862bb05 | 9 | 1 | rebalance_test_table_0
--------------------------------------------------
Location: /B
--------------------------------------------------
Per-server replica distribution summary:
Statistic | Value
-----------------------+----------
Minimum Replica Count | 2
Maximum Replica Count | 3
Average Replica Count | 2.666667
Per-server replica distribution details:
UUID | Address | Replica Count
----------------------------------+---------------------+---------------
9ddba97e3a4f4b4b99e67a34c4a0e3ec | 127.14.19.134:38075 | 2
856267e243cb46ab884b7a1694ae465e | 127.14.19.132:40673 | 3
c2867695ca20469dba7d61ebc5ecebfa | 127.14.19.133:36633 | 3
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 0
Maximum | 1
Average | 0.333333
Per-table replica distribution details for tables:
Table Id | Replica Count | Replica Skew | Table Name
----------------------------------+---------------+--------------+------------------------
16527c7f48bd4f6b9d141bb2928a5aa6 | 9 | 0 | rebalance_test_table_2
924c92d998df4505a2cf1395cf34abc3 | 9 | 0 | rebalance_test_table_1
b8710a737ffd4f5f93b9e9df5862bb05 | 9 | 1 | rebalance_test_table_0
Placement policy violations:
Location | Number of non-complying tables | Number of non-complying tablets
----------+--------------------------------+---------------------------------
/A | 3 | 9
Placement policy violation details:
Location | Table Name | Tablet | RF | Replicas at location
----------+------------------------+----------------------------------+----+----------------------
/A | rebalance_test_table_2 | a4054e1b6e654bc4b57d42be2a85296e | 3 | 2
/A | rebalance_test_table_2 | eefff01740a34c40ba687c7242b87770 | 3 | 2
/A | rebalance_test_table_0 | 6fde79de661a4993ace77eadfc877bf4 | 3 | 3
/A | rebalance_test_table_1 | 48cdbe84b6094cccaf69b553fd8f1d68 | 3 | 2
/A | rebalance_test_table_0 | 0a02da6c3e2d4db5b1db2f1de6824bc5 | 3 | 2
/A | rebalance_test_table_2 | d0eee4c5c9b54dfca9b59276787773db | 3 | 2
/A | rebalance_test_table_1 | d68741aa20be48229a19dda143743bca | 3 | 2
/A | rebalance_test_table_0 | b373739a50754ecb8753a84d813c0151 | 3 | 2
/A | rebalance_test_table_1 | 56aea05a8a714b4ca357e24049c7f348 | 3 | 2
stderr: W20250110 03:07:45.545210 2922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:45.545356 2922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:45.546480 2922 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250110 03:07:45.664688 2922 placement_policy_util.cc:407] tablet b373739a50754ecb8753a84d813c0151: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664744 2922 placement_policy_util.cc:407] tablet 0a02da6c3e2d4db5b1db2f1de6824bc5: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664765 2922 placement_policy_util.cc:407] tablet 6fde79de661a4993ace77eadfc877bf4: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664785 2922 placement_policy_util.cc:407] tablet 48cdbe84b6094cccaf69b553fd8f1d68: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664809 2922 placement_policy_util.cc:407] tablet eefff01740a34c40ba687c7242b87770: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664834 2922 placement_policy_util.cc:407] tablet 56aea05a8a714b4ca357e24049c7f348: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664857 2922 placement_policy_util.cc:407] tablet a4054e1b6e654bc4b57d42be2a85296e: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664880 2922 placement_policy_util.cc:407] tablet d68741aa20be48229a19dda143743bca: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.664923 2922 placement_policy_util.cc:407] tablet d0eee4c5c9b54dfca9b59276787773db: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.665182 2922 rebalancer_tool.cc:237] fixing placement policy violations
I20250110 03:07:45.718056 2922 placement_policy_util.cc:407] tablet b373739a50754ecb8753a84d813c0151: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718102 2922 placement_policy_util.cc:407] tablet 0a02da6c3e2d4db5b1db2f1de6824bc5: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718115 2922 placement_policy_util.cc:407] tablet a4054e1b6e654bc4b57d42be2a85296e: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718132 2922 placement_policy_util.cc:407] tablet eefff01740a34c40ba687c7242b87770: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718151 2922 placement_policy_util.cc:407] tablet d68741aa20be48229a19dda143743bca: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718173 2922 placement_policy_util.cc:407] tablet d0eee4c5c9b54dfca9b59276787773db: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718189 2922 placement_policy_util.cc:407] tablet 6fde79de661a4993ace77eadfc877bf4: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718205 2922 placement_policy_util.cc:407] tablet 48cdbe84b6094cccaf69b553fd8f1d68: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718228 2922 placement_policy_util.cc:407] tablet 56aea05a8a714b4ca357e24049c7f348: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:45.718415 2922 rebalancer_tool.cc:246] running cross-location rebalancing
W20250110 03:07:45.759467 2922 rebalancer_tool.cc:1221] No replica could be moved this iteration: Not found: table b8710a737ffd4f5f93b9e9df5862bb05: could not find any suitable replica to move from server 84c4433aa11f4e2f9a49bdb01aba84e0 to server 856267e243cb46ab884b7a1694ae465e
I20250110 03:07:45.767143 2922 rebalancer_tool.cc:1090] tablet 0a02da6c3e2d4db5b1db2f1de6824bc5: '84c4433aa11f4e2f9a49bdb01aba84e0' -> '856267e243cb46ab884b7a1694ae465e' move scheduled
I20250110 03:07:45.773545 2922 rebalancer_tool.cc:1090] tablet 48cdbe84b6094cccaf69b553fd8f1d68: '90d44e46d37f48258cea315fd7e589d5' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move scheduled
I20250110 03:07:45.780560 2922 rebalancer_tool.cc:1090] tablet eefff01740a34c40ba687c7242b87770: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> 'c2867695ca20469dba7d61ebc5ecebfa' move scheduled
I20250110 03:07:45.786072 2922 rebalancer_tool.cc:1090] tablet a4054e1b6e654bc4b57d42be2a85296e: '84c4433aa11f4e2f9a49bdb01aba84e0' -> '856267e243cb46ab884b7a1694ae465e' move scheduled
I20250110 03:07:45.792323 2922 rebalancer_tool.cc:1090] tablet d68741aa20be48229a19dda143743bca: '90d44e46d37f48258cea315fd7e589d5' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move scheduled
I20250110 03:07:45.798491 2922 rebalancer_tool.cc:1090] tablet b373739a50754ecb8753a84d813c0151: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> 'c2867695ca20469dba7d61ebc5ecebfa' move scheduled
I20250110 03:07:45.804580 2922 rebalancer_tool.cc:1090] tablet 56aea05a8a714b4ca357e24049c7f348: '90d44e46d37f48258cea315fd7e589d5' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move scheduled
I20250110 03:07:45.810518 2922 rebalancer_tool.cc:1090] tablet d0eee4c5c9b54dfca9b59276787773db: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> 'c2867695ca20469dba7d61ebc5ecebfa' move scheduled
I20250110 03:07:47.169356 2922 rebalancer_tool.cc:1152] tablet 56aea05a8a714b4ca357e24049c7f348: '90d44e46d37f48258cea315fd7e589d5' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move completed: OK
I20250110 03:07:47.418816 2922 rebalancer_tool.cc:1152] tablet 48cdbe84b6094cccaf69b553fd8f1d68: '90d44e46d37f48258cea315fd7e589d5' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move completed: OK
I20250110 03:07:47.421998 2922 rebalancer_tool.cc:1152] tablet 0a02da6c3e2d4db5b1db2f1de6824bc5: '84c4433aa11f4e2f9a49bdb01aba84e0' -> '856267e243cb46ab884b7a1694ae465e' move completed: OK
I20250110 03:07:47.437166 2922 rebalancer_tool.cc:1152] tablet d68741aa20be48229a19dda143743bca: '90d44e46d37f48258cea315fd7e589d5' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move completed: OK
I20250110 03:07:47.680235 2922 rebalancer_tool.cc:1152] tablet a4054e1b6e654bc4b57d42be2a85296e: '84c4433aa11f4e2f9a49bdb01aba84e0' -> '856267e243cb46ab884b7a1694ae465e' move completed: OK
I20250110 03:07:47.683394 2922 rebalancer_tool.cc:1152] tablet b373739a50754ecb8753a84d813c0151: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> 'c2867695ca20469dba7d61ebc5ecebfa' move completed: OK
I20250110 03:07:48.313534 2922 rebalancer_tool.cc:1152] tablet eefff01740a34c40ba687c7242b87770: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> 'c2867695ca20469dba7d61ebc5ecebfa' move completed: OK
I20250110 03:07:48.524416 2922 rebalancer_tool.cc:1152] tablet d0eee4c5c9b54dfca9b59276787773db: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> 'c2867695ca20469dba7d61ebc5ecebfa' move completed: OK
W20250110 03:07:48.566890 2922 rebalancer_tool.cc:1221] No replica could be moved this iteration: Not found: table b8710a737ffd4f5f93b9e9df5862bb05: could not find any suitable replica to move from server 84c4433aa11f4e2f9a49bdb01aba84e0 to server 856267e243cb46ab884b7a1694ae465e
I20250110 03:07:48.567108 2922 rebalancer_tool.cc:283] starting rebalancing within location '/B'
I20250110 03:07:48.567210 2922 rebalancer_tool.cc:283] starting rebalancing within location '/A'
I20250110 03:07:48.619593 3729 rebalancer_tool.cc:1090] tablet d68741aa20be48229a19dda143743bca: '84c4433aa11f4e2f9a49bdb01aba84e0' -> '90d44e46d37f48258cea315fd7e589d5' move scheduled
I20250110 03:07:48.622932 3728 rebalancer_tool.cc:1090] tablet d68741aa20be48229a19dda143743bca: '9ddba97e3a4f4b4b99e67a34c4a0e3ec' -> '856267e243cb46ab884b7a1694ae465e' move scheduled
I20250110 03:07:48.628448 3729 rebalancer_tool.cc:1090] tablet 48cdbe84b6094cccaf69b553fd8f1d68: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> '90d44e46d37f48258cea315fd7e589d5' move scheduled
I20250110 03:07:48.632119 3728 rebalancer_tool.cc:1090] tablet d0eee4c5c9b54dfca9b59276787773db: 'c2867695ca20469dba7d61ebc5ecebfa' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move scheduled
I20250110 03:07:48.637259 3729 rebalancer_tool.cc:1090] tablet eefff01740a34c40ba687c7242b87770: '90d44e46d37f48258cea315fd7e589d5' -> 'baa5f3e8148b4fdda964dbc1cb4a30c0' move scheduled
I20250110 03:07:48.642117 3728 rebalancer_tool.cc:1090] tablet 56aea05a8a714b4ca357e24049c7f348: '9ddba97e3a4f4b4b99e67a34c4a0e3ec' -> 'c2867695ca20469dba7d61ebc5ecebfa' move scheduled
I20250110 03:07:50.137887 3728 rebalancer_tool.cc:1152] tablet 56aea05a8a714b4ca357e24049c7f348: '9ddba97e3a4f4b4b99e67a34c4a0e3ec' -> 'c2867695ca20469dba7d61ebc5ecebfa' move completed: OK
I20250110 03:07:50.141721 3729 rebalancer_tool.cc:1152] tablet d68741aa20be48229a19dda143743bca: '84c4433aa11f4e2f9a49bdb01aba84e0' -> '90d44e46d37f48258cea315fd7e589d5' move completed: OK
I20250110 03:07:50.141764 3728 rebalancer_tool.cc:1152] tablet d68741aa20be48229a19dda143743bca: '9ddba97e3a4f4b4b99e67a34c4a0e3ec' -> '856267e243cb46ab884b7a1694ae465e' move completed: OK
I20250110 03:07:50.145742 3729 rebalancer_tool.cc:1152] tablet 48cdbe84b6094cccaf69b553fd8f1d68: 'baa5f3e8148b4fdda964dbc1cb4a30c0' -> '90d44e46d37f48258cea315fd7e589d5' move completed: OK
I20250110 03:07:50.356310 3728 rebalancer_tool.cc:1152] tablet d0eee4c5c9b54dfca9b59276787773db: 'c2867695ca20469dba7d61ebc5ecebfa' -> '9ddba97e3a4f4b4b99e67a34c4a0e3ec' move completed: OK
I20250110 03:07:50.357275 3729 rebalancer_tool.cc:1152] tablet eefff01740a34c40ba687c7242b87770: '90d44e46d37f48258cea315fd7e589d5' -> 'baa5f3e8148b4fdda964dbc1cb4a30c0' move completed: OK
I20250110 03:07:50.441593 2922 placement_policy_util.cc:407] tablet a4054e1b6e654bc4b57d42be2a85296e: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441628 2922 placement_policy_util.cc:407] tablet eefff01740a34c40ba687c7242b87770: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441641 2922 placement_policy_util.cc:407] tablet 6fde79de661a4993ace77eadfc877bf4: detected majority of replicas (3 of 3) at location /A
I20250110 03:07:50.441653 2922 placement_policy_util.cc:407] tablet 48cdbe84b6094cccaf69b553fd8f1d68: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441664 2922 placement_policy_util.cc:407] tablet 0a02da6c3e2d4db5b1db2f1de6824bc5: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441676 2922 placement_policy_util.cc:407] tablet d0eee4c5c9b54dfca9b59276787773db: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441687 2922 placement_policy_util.cc:407] tablet d68741aa20be48229a19dda143743bca: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441699 2922 placement_policy_util.cc:407] tablet b373739a50754ecb8753a84d813c0151: detected majority of replicas (2 of 3) at location /A
I20250110 03:07:50.441709 2922 placement_policy_util.cc:407] tablet 56aea05a8a714b4ca357e24049c7f348: detected majority of replicas (2 of 3) at location /A
*** Aborted at 1736478470 (unix time) try "date -d @1736478470" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGSEGV (@0x1000055e152) received by PID 2922 (TID 0x7f332037a480) from PID 5628242; stack trace: ***
@ 0x7f3328c3d980 (unknown) at ??:0
@ 0x7f332836d873 tcmalloc::ThreadCache::ReleaseToCentralCache() at ??:0
@ 0x7f332836dbe7 tcmalloc::ThreadCache::Scavenge() at ??:0
@ 0x7f3326503271 OPENSSL_LH_free at ??:0
@ 0x7f33264e1bfd (unknown) at ??:0
@ 0x7f3326500e10 OPENSSL_cleanup at ??:0
@ 0x7f3326c84161 (unknown) at ??:0
@ 0x7f3326c8425a exit at ??:0
@ 0x7f3326c62bfe __libc_start_main at ??:0
@ 0x55e12db6df5a _start at ??:0
I20250110 03:07:50.593336 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 1537
I20250110 03:07:50.599345 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 1668
I20250110 03:07:50.605386 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 1801
I20250110 03:07:50.612234 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 2791
I20250110 03:07:50.617363 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 2660
I20250110 03:07:50.622309 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 2529
I20250110 03:07:50.627511 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 1446
2025-01-10T03:07:50Z chronyd exiting
I20250110 03:07:50.643268 14414 test_util.cc:181] -----------------------------------------------
I20250110 03:07:50.643306 14414 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1736478394679828-14414-0
[ FAILED ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3, where GetParam() = { rep_factor: 3, num_tables: 3, location_info: { /A:3 /B:3 } } (11540 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20250110 03:07:50.643671 14414 test_util.cc:274] Using random seed: 1718034388
I20250110 03:07:50.644124 14414 ts_itest-base.cc:112] Starting cluster with:
I20250110 03:07:50.644152 14414 ts_itest-base.cc:113] --------------
I20250110 03:07:50.644179 14414 ts_itest-base.cc:114] 5 tablet servers
I20250110 03:07:50.644207 14414 ts_itest-base.cc:115] 3 replicas per TS
I20250110 03:07:50.644232 14414 ts_itest-base.cc:116] --------------
2025-01-10T03:07:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T03:07:50Z Disabled control of system clock
I20250110 03:07:50.650971 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:39657
--webserver_interface=127.14.19.190
--webserver_port=0
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.19.190:39657
--location_mapping_cmd=/tmp/dist-test-taskWQrDfu/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20250110 03:07:50.723565 4209 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20250110 03:07:50.723732 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:50.723757 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:50.723783 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:50.725016 4209 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:50.725059 4209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 03:07:50.725102 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:50.725128 4209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 03:07:50.725149 4209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 03:07:50.726580 4209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskWQrDfu/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.14.19.190:39657
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.19.190:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.14.19.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:50.726786 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:50.727025 4209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:50.730724 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:50.731869 4209 server_base.cc:1034] running on GCE node
W20250110 03:07:50.732826 4214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:50.733659 4215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:50.733906 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:50.734120 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:50.735249 4209 hybrid_clock.cc:648] HybridClock initialized: now 1736478470735238 us; error 37 us; skew 500 ppm
I20250110 03:07:50.736358 4209 webserver.cc:458] Webserver started at http://127.14.19.190:37963/ using document root <none> and password file <none>
I20250110 03:07:50.736518 4209 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:50.736557 4209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:50.736624 4209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:50.737396 4209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d70d31796827461f89fedd8ef9943800"
format_stamp: "Formatted at 2025-01-10 03:07:50 on dist-test-slave-nrvn"
I20250110 03:07:50.737656 4209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d70d31796827461f89fedd8ef9943800"
format_stamp: "Formatted at 2025-01-10 03:07:50 on dist-test-slave-nrvn"
I20250110 03:07:50.738756 4209 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250110 03:07:50.739387 4223 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:50.739567 4209 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:50.739634 4209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d70d31796827461f89fedd8ef9943800"
format_stamp: "Formatted at 2025-01-10 03:07:50 on dist-test-slave-nrvn"
I20250110 03:07:50.739681 4209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:50.755072 4209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:50.755298 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:50.755379 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:50.758693 4209 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.190:39657
I20250110 03:07:50.758719 4275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.190:39657 every 8 connection(s)
I20250110 03:07:50.759002 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 03:07:50.759502 4276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:50.761618 4276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800: Bootstrap starting.
I20250110 03:07:50.762130 4276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:50.762333 4276 log.cc:826] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:50.762847 4276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800: No bootstrap required, opened a new log
I20250110 03:07:50.764187 4276 raft_consensus.cc:357] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70d31796827461f89fedd8ef9943800" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 39657 } }
I20250110 03:07:50.764287 4276 raft_consensus.cc:383] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:50.764307 4276 raft_consensus.cc:738] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70d31796827461f89fedd8ef9943800, State: Initialized, Role: FOLLOWER
I20250110 03:07:50.764394 4276 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70d31796827461f89fedd8ef9943800" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 39657 } }
I20250110 03:07:50.764458 4276 raft_consensus.cc:397] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 03:07:50.764487 4276 raft_consensus.cc:491] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 03:07:50.764510 4276 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:50.764577 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 4209
I20250110 03:07:50.764660 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 03:07:50.764966 4276 raft_consensus.cc:513] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70d31796827461f89fedd8ef9943800" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 39657 } }
I20250110 03:07:50.765091 4276 leader_election.cc:304] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70d31796827461f89fedd8ef9943800; no voters:
I20250110 03:07:50.765266 4276 leader_election.cc:290] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 03:07:50.765328 4279 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:50.765451 4279 raft_consensus.cc:695] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [term 1 LEADER]: Becoming Leader. State: Replica: d70d31796827461f89fedd8ef9943800, State: Running, Role: LEADER
I20250110 03:07:50.765491 4276 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 03:07:50.765534 4279 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70d31796827461f89fedd8ef9943800" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 39657 } }
I20250110 03:07:50.766201 4283 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70d31796827461f89fedd8ef9943800. Latest consensus state: current_term: 1 leader_uuid: "d70d31796827461f89fedd8ef9943800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70d31796827461f89fedd8ef9943800" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 39657 } } }
I20250110 03:07:50.766330 4283 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [sys.catalog]: This master's current role is: LEADER
I20250110 03:07:50.766232 4282 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70d31796827461f89fedd8ef9943800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70d31796827461f89fedd8ef9943800" member_type: VOTER last_known_addr { host: "127.14.19.190" port: 39657 } } }
I20250110 03:07:50.766492 4282 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800 [sys.catalog]: This master's current role is: LEADER
W20250110 03:07:50.767398 4296 catalog_manager.cc:1559] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 03:07:50.767441 4296 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 03:07:50.767509 4290 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 03:07:50.767661 4290 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 03:07:50.769182 4290 catalog_manager.cc:1348] Generated new cluster ID: f7558fa3b11e46dfa04fd02eef6532f9
I20250110 03:07:50.769239 4290 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 03:07:50.781136 4290 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 03:07:50.781616 4290 catalog_manager.cc:1505] Loading token signing keys...
I20250110 03:07:50.797566 4290 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d70d31796827461f89fedd8ef9943800: Generated new TSK 0
I20250110 03:07:50.797719 4290 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 03:07:50.805153 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.129:0
--local_ip_for_outbound_sockets=127.14.19.129
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:50.883734 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:50.883908 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:50.883939 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:50.885211 4300 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:50.885251 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:50.885288 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.129
I20250110 03:07:50.886690 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.14.19.129
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.129
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:50.886904 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:50.887121 4300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:50.892419 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:50.893249 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:50.893482 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:50.894409 4300 server_base.cc:1034] running on GCE node
I20250110 03:07:50.894600 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:50.894790 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:50.895924 4300 hybrid_clock.cc:648] HybridClock initialized: now 1736478470895906 us; error 38 us; skew 500 ppm
I20250110 03:07:50.897128 4300 webserver.cc:458] Webserver started at http://127.14.19.129:33779/ using document root <none> and password file <none>
I20250110 03:07:50.897301 4300 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:50.897348 4300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:50.897440 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:50.898308 4300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e3a520ce69d74074a3d309fd843bccd6"
format_stamp: "Formatted at 2025-01-10 03:07:50 on dist-test-slave-nrvn"
I20250110 03:07:50.898653 4300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e3a520ce69d74074a3d309fd843bccd6"
format_stamp: "Formatted at 2025-01-10 03:07:50 on dist-test-slave-nrvn"
I20250110 03:07:50.899788 4300 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:50.900399 4314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:50.900575 4300 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:50.900640 4300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e3a520ce69d74074a3d309fd843bccd6"
format_stamp: "Formatted at 2025-01-10 03:07:50 on dist-test-slave-nrvn"
I20250110 03:07:50.900698 4300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:50.914424 4300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:50.914659 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:50.914753 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:50.914944 4300 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:50.915220 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:50.915251 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:50.915282 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:50.915308 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:50.921382 4300 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.129:37651
I20250110 03:07:50.921422 4427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.129:37651 every 8 connection(s)
I20250110 03:07:50.921666 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 03:07:50.925824 4428 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:50.925902 4428 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:50.926059 4428 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:50.929857 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 4300
I20250110 03:07:50.929955 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 03:07:50.931461 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.130:0
--local_ip_for_outbound_sockets=127.14.19.130
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:50.958071 4240 ts_manager.cc:194] Registered new tserver with Master: e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:50.959156 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.129:35841
W20250110 03:07:51.005283 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:51.005410 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:51.005429 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:51.006525 4432 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:51.006573 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:51.006594 4432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.130
I20250110 03:07:51.007870 4432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.14.19.130
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.130
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:51.008018 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:51.008174 4432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:51.012079 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.013090 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.014264 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:51.015044 4432 server_base.cc:1034] running on GCE node
I20250110 03:07:51.015178 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:51.015329 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:51.016448 4432 hybrid_clock.cc:648] HybridClock initialized: now 1736478471016431 us; error 34 us; skew 500 ppm
I20250110 03:07:51.017602 4432 webserver.cc:458] Webserver started at http://127.14.19.130:35019/ using document root <none> and password file <none>
I20250110 03:07:51.017778 4432 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:51.017835 4432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:51.017931 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:51.018754 4432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.019034 4432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.020099 4432 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:51.020699 4446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.020851 4432 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:51.020923 4432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.020996 4432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:51.033721 4432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:51.033927 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:51.034027 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:51.034173 4432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:51.034430 4432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:51.034458 4432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.034489 4432 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:51.034520 4432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.039729 4432 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.130:43399
I20250110 03:07:51.039783 4559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.130:43399 every 8 connection(s)
I20250110 03:07:51.040020 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 03:07:51.044546 4560 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:51.044627 4560 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:51.044823 4560 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:51.045359 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 4432
I20250110 03:07:51.045451 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 03:07:51.047770 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.131:0
--local_ip_for_outbound_sockets=127.14.19.131
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:51.076188 4240 ts_manager.cc:194] Registered new tserver with Master: e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:51.076668 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.130:47361
W20250110 03:07:51.121565 4564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:51.121716 4564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:51.121745 4564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:51.123006 4564 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:51.123045 4564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:51.123080 4564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.131
I20250110 03:07:51.124423 4564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.14.19.131
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.4564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.131
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:51.124641 4564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:51.124832 4564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:51.129277 4572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.129451 4569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.130523 4570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:51.131320 4564 server_base.cc:1034] running on GCE node
I20250110 03:07:51.131441 4564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:51.131592 4564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:51.132710 4564 hybrid_clock.cc:648] HybridClock initialized: now 1736478471132705 us; error 35 us; skew 500 ppm
I20250110 03:07:51.133742 4564 webserver.cc:458] Webserver started at http://127.14.19.131:43581/ using document root <none> and password file <none>
I20250110 03:07:51.133895 4564 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:51.133935 4564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:51.134001 4564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:51.134819 4564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b680ae46493d4601a57d0545c65a1895"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.135074 4564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b680ae46493d4601a57d0545c65a1895"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.136163 4564 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:51.136734 4578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.136912 4564 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:51.136970 4564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b680ae46493d4601a57d0545c65a1895"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.137012 4564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:51.156365 4564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:51.156529 4564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:51.156605 4564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:51.156735 4564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:51.156991 4564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:51.157022 4564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.157042 4564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:51.157068 4564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.162083 4564 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.131:40061
I20250110 03:07:51.162147 4691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.131:40061 every 8 connection(s)
I20250110 03:07:51.162369 4564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 03:07:51.166400 4692 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:51.166473 4692 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:51.166668 4692 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:51.171810 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 4564
I20250110 03:07:51.171890 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 03:07:51.173017 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:0
--local_ip_for_outbound_sockets=127.14.19.132
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:51.198364 4240 ts_manager.cc:194] Registered new tserver with Master: b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:51.198868 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.131:41635
W20250110 03:07:51.246665 4696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:51.246860 4696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:51.246891 4696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:51.248097 4696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:51.248136 4696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:51.248171 4696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:51.249508 4696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.4696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:51.249737 4696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:51.249943 4696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:51.253819 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.254616 4704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.255314 4702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:51.256433 4696 server_base.cc:1034] running on GCE node
I20250110 03:07:51.256573 4696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:51.256718 4696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:51.257838 4696 hybrid_clock.cc:648] HybridClock initialized: now 1736478471257816 us; error 33 us; skew 500 ppm
I20250110 03:07:51.258934 4696 webserver.cc:458] Webserver started at http://127.14.19.132:41275/ using document root <none> and password file <none>
I20250110 03:07:51.259114 4696 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:51.259161 4696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:51.259251 4696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:51.260083 4696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "22901baa49564439bfd6978fe2e4a4a9"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.260367 4696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "22901baa49564439bfd6978fe2e4a4a9"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.261482 4696 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20250110 03:07:51.262063 4710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.262219 4696 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.262287 4696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "22901baa49564439bfd6978fe2e4a4a9"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.262343 4696 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:51.271848 4696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:51.272028 4696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:51.272127 4696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:51.272274 4696 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:51.272521 4696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:51.272552 4696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.272584 4696 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:51.272616 4696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.277818 4696 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:41431
I20250110 03:07:51.277870 4823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:41431 every 8 connection(s)
I20250110 03:07:51.278116 4696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:51.282460 4824 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:51.282531 4824 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:51.282728 4824 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:51.286775 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 4696
I20250110 03:07:51.286852 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 03:07:51.287868 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:0
--local_ip_for_outbound_sockets=127.14.19.133
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:51.322918 4240 ts_manager.cc:194] Registered new tserver with Master: 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431)
I20250110 03:07:51.323348 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:49237
W20250110 03:07:51.370095 4828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:51.370227 4828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:51.370247 4828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:51.371562 4828 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:51.371618 4828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:51.371657 4828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:07:51.373365 4828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=0
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.4828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:51.373555 4828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:51.373771 4828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:51.377573 4833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.378432 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:51.379300 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:51.380362 4828 server_base.cc:1034] running on GCE node
I20250110 03:07:51.380527 4828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:51.380709 4828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:51.381836 4828 hybrid_clock.cc:648] HybridClock initialized: now 1736478471381821 us; error 26 us; skew 500 ppm
I20250110 03:07:51.382923 4828 webserver.cc:458] Webserver started at http://127.14.19.133:39041/ using document root <none> and password file <none>
I20250110 03:07:51.383118 4828 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:51.383167 4828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:51.383255 4828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 03:07:51.384039 4828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8cb2e4afb3384e40879d605ea471b039"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.384334 4828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8cb2e4afb3384e40879d605ea471b039"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.385376 4828 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20250110 03:07:51.385923 4842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.386066 4828 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.386138 4828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8cb2e4afb3384e40879d605ea471b039"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:51.386193 4828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:51.415331 4828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:51.415525 4828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:51.415627 4828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:51.415787 4828 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:51.416060 4828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:51.416097 4828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.416129 4828 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:51.416154 4828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:51.421228 4828 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:34025
I20250110 03:07:51.421293 4955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:34025 every 8 connection(s)
I20250110 03:07:51.421530 4828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:07:51.423529 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 4828
I20250110 03:07:51.423627 14414 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 03:07:51.426017 4956 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:51.426093 4956 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:51.426232 4956 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:51.460875 4240 ts_manager.cc:194] Registered new tserver with Master: 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025)
I20250110 03:07:51.461315 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:46093
I20250110 03:07:51.467433 14414 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 03:07:51.473044 14414 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 03:07:51.473106 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 4828
I20250110 03:07:51.477317 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 4696
I20250110 03:07:51.960124 4428 heartbeater.cc:502] Master 127.14.19.190:39657 was elected leader, sending a full tablet report...
I20250110 03:07:52.077540 4560 heartbeater.cc:502] Master 127.14.19.190:39657 was elected leader, sending a full tablet report...
I20250110 03:07:52.199780 4692 heartbeater.cc:502] Master 127.14.19.190:39657 was elected leader, sending a full tablet report...
I20250110 03:07:55.232288 4240 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51554:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250110 03:07:55.233986 4240 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 03:07:55.240526 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 56330dd8a2ef4597be2d809c1cf02000 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.240695 4492 tablet_service.cc:1467] Processing CreateTablet for tablet 8953c1791507457e805dd8a89638dd29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.240829 4492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8953c1791507457e805dd8a89638dd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.240521 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.241079 4494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.241252 4493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56330dd8a2ef4597be2d809c1cf02000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.241175 4626 tablet_service.cc:1467] Processing CreateTablet for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.241583 4626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.242123 4624 tablet_service.cc:1467] Processing CreateTablet for tablet 8953c1791507457e805dd8a89638dd29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.242238 4624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8953c1791507457e805dd8a89638dd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.242462 4625 tablet_service.cc:1467] Processing CreateTablet for tablet 56330dd8a2ef4597be2d809c1cf02000 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.242442 4362 tablet_service.cc:1467] Processing CreateTablet for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.242570 4625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56330dd8a2ef4597be2d809c1cf02000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.242719 4362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.244606 4976 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:55.244778 4360 tablet_service.cc:1467] Processing CreateTablet for tablet 8953c1791507457e805dd8a89638dd29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.244884 4360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8953c1791507457e805dd8a89638dd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.245281 4976 tablet_bootstrap.cc:654] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.245513 4977 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:55.245532 4976 log.cc:826] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:55.246039 4361 tablet_service.cc:1467] Processing CreateTablet for tablet 56330dd8a2ef4597be2d809c1cf02000 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.246055 4977 tablet_bootstrap.cc:654] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.246157 4361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56330dd8a2ef4597be2d809c1cf02000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.246192 4976 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:55.246266 4976 ts_tablet_manager.cc:1397] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250110 03:07:55.246416 4977 log.cc:826] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:55.247318 4977 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:55.247370 4977 ts_tablet_manager.cc:1397] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20250110 03:07:55.247741 4976 raft_consensus.cc:357] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.247850 4976 raft_consensus.cc:383] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.247884 4976 raft_consensus.cc:738] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.248016 4976 consensus_queue.cc:260] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.248239 4976 ts_tablet_manager.cc:1428] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 03:07:55.248355 4976 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:55.248764 4976 tablet_bootstrap.cc:654] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.249114 4977 raft_consensus.cc:357] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.249243 4977 raft_consensus.cc:383] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.249271 4977 raft_consensus.cc:738] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.249339 4977 consensus_queue.cc:260] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.249565 4977 ts_tablet_manager.cc:1428] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250110 03:07:55.249564 4975 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:55.249605 4976 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:55.249640 4976 ts_tablet_manager.cc:1397] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:55.249671 4977 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:55.249784 4976 raft_consensus.cc:357] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.249840 4976 raft_consensus.cc:383] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.249864 4976 raft_consensus.cc:738] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.249917 4976 consensus_queue.cc:260] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.250084 4977 tablet_bootstrap.cc:654] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.250128 4975 tablet_bootstrap.cc:654] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.250250 4976 ts_tablet_manager.cc:1428] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 03:07:55.250313 4976 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:55.250399 4975 log.cc:826] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:55.250744 4976 tablet_bootstrap.cc:654] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.250982 4977 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:55.251124 4977 ts_tablet_manager.cc:1397] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.251237 4975 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:55.251300 4976 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:55.251302 4975 ts_tablet_manager.cc:1397] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:55.251344 4976 ts_tablet_manager.cc:1397] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.251313 4977 raft_consensus.cc:357] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.251376 4977 raft_consensus.cc:383] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.251399 4977 raft_consensus.cc:738] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.251441 4977 consensus_queue.cc:260] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.251461 4976 raft_consensus.cc:357] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.251513 4976 raft_consensus.cc:383] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.251518 4977 ts_tablet_manager.cc:1428] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.251541 4976 raft_consensus.cc:738] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.251574 4977 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:55.251576 4976 consensus_queue.cc:260] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.251847 4976 ts_tablet_manager.cc:1428] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.252084 4977 tablet_bootstrap.cc:654] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.252758 4977 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:55.252810 4977 ts_tablet_manager.cc:1397] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.252933 4977 raft_consensus.cc:357] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.253005 4977 raft_consensus.cc:383] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.253029 4977 raft_consensus.cc:738] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.253087 4977 consensus_queue.cc:260] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.253317 4977 ts_tablet_manager.cc:1428] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.253535 4975 raft_consensus.cc:357] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.253660 4975 raft_consensus.cc:383] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.253702 4975 raft_consensus.cc:738] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.253784 4975 consensus_queue.cc:260] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.253997 4975 ts_tablet_manager.cc:1428] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250110 03:07:55.254067 4975 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:55.254410 4975 tablet_bootstrap.cc:654] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.254966 4975 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:55.255029 4975 ts_tablet_manager.cc:1397] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.255168 4975 raft_consensus.cc:357] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.255226 4975 raft_consensus.cc:383] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.255249 4975 raft_consensus.cc:738] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.255293 4975 consensus_queue.cc:260] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.255366 4975 ts_tablet_manager.cc:1428] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.255414 4975 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:55.255750 4975 tablet_bootstrap.cc:654] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.256276 4975 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:55.256314 4975 ts_tablet_manager.cc:1397] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.256433 4975 raft_consensus.cc:357] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.256482 4975 raft_consensus.cc:383] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.256495 4975 raft_consensus.cc:738] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.256526 4975 consensus_queue.cc:260] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.256728 4975 ts_tablet_manager.cc:1428] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20250110 03:07:55.292595 4561 tablet.cc:2367] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:55.295804 4980 raft_consensus.cc:491] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:55.295876 4980 raft_consensus.cc:513] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.296131 4980 leader_election.cc:290] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:55.299054 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953c1791507457e805dd8a89638dd29" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" is_pre_election: true
I20250110 03:07:55.299160 4514 raft_consensus.cc:2463] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:55.299328 4579 leader_election.cc:304] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.299435 4980 raft_consensus.cc:2798] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:55.299485 4980 raft_consensus.cc:491] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:55.299510 4980 raft_consensus.cc:3054] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.300100 4980 raft_consensus.cc:513] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.300197 4980 leader_election.cc:290] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Requested vote from peers e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:55.300366 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953c1791507457e805dd8a89638dd29" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:07:55.300443 4514 raft_consensus.cc:3054] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.300518 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953c1791507457e805dd8a89638dd29" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:55.300519 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953c1791507457e805dd8a89638dd29" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:55.300673 4382 raft_consensus.cc:2463] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:55.301151 4514 raft_consensus.cc:2463] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:55.301299 4579 leader_election.cc:304] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters: e3a520ce69d74074a3d309fd843bccd6
I20250110 03:07:55.301390 4980 raft_consensus.cc:2798] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:55.301501 4980 raft_consensus.cc:695] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Becoming Leader. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Running, Role: LEADER
I20250110 03:07:55.301589 4980 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.302289 4239 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 reported cstate change: term changed from 0 to 1, leader changed from <none> to b680ae46493d4601a57d0545c65a1895 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } }
W20250110 03:07:55.415732 4693 tablet.cc:2367] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 03:07:55.424532 4429 tablet.cc:2367] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 03:07:55.458977 4987 raft_consensus.cc:491] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:55.459054 4987 raft_consensus.cc:513] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.459316 4987 leader_election.cc:290] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061), e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:55.462070 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78c4d4f4de4b83b2b63a3b31e20fae" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895" is_pre_election: true
I20250110 03:07:55.462208 4646 raft_consensus.cc:2463] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 0.
I20250110 03:07:55.462181 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78c4d4f4de4b83b2b63a3b31e20fae" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:55.462267 4382 raft_consensus.cc:2463] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 0.
I20250110 03:07:55.462433 4448 leader_election.cc:304] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.462599 4987 raft_consensus.cc:2798] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:55.462661 4987 raft_consensus.cc:491] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:55.462692 4987 raft_consensus.cc:3054] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.463344 4987 raft_consensus.cc:513] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.463457 4987 leader_election.cc:290] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 election: Requested vote from peers b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061), e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:55.463652 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78c4d4f4de4b83b2b63a3b31e20fae" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895"
I20250110 03:07:55.463718 4646 raft_consensus.cc:3054] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.463752 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f78c4d4f4de4b83b2b63a3b31e20fae" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:55.463824 4382 raft_consensus.cc:3054] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.464499 4646 raft_consensus.cc:2463] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 1.
I20250110 03:07:55.464592 4382 raft_consensus.cc:2463] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 1.
I20250110 03:07:55.464643 4448 leader_election.cc:304] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.464717 4987 raft_consensus.cc:2798] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:55.464840 4987 raft_consensus.cc:695] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Becoming Leader. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Running, Role: LEADER
I20250110 03:07:55.464926 4987 consensus_queue.cc:237] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:07:55.465600 4239 catalog_manager.cc:5526] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:55.562585 4987 raft_consensus.cc:491] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:55.562676 4987 raft_consensus.cc:513] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.562865 4987 leader_election.cc:290] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:55.563107 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56330dd8a2ef4597be2d809c1cf02000" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895" is_pre_election: true
I20250110 03:07:55.563135 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56330dd8a2ef4597be2d809c1cf02000" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:55.563194 4646 raft_consensus.cc:2463] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 0.
I20250110 03:07:55.563225 4382 raft_consensus.cc:2463] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 0.
I20250110 03:07:55.563339 4448 leader_election.cc:304] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.563441 4987 raft_consensus.cc:2798] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:55.563484 4987 raft_consensus.cc:491] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:55.563510 4987 raft_consensus.cc:3054] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.564103 4987 raft_consensus.cc:513] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.564225 4987 leader_election.cc:290] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 election: Requested vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:55.564373 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56330dd8a2ef4597be2d809c1cf02000" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895"
I20250110 03:07:55.564373 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56330dd8a2ef4597be2d809c1cf02000" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:55.564447 4646 raft_consensus.cc:3054] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.564447 4382 raft_consensus.cc:3054] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.565099 4382 raft_consensus.cc:2463] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 1.
I20250110 03:07:55.565109 4646 raft_consensus.cc:2463] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 1.
I20250110 03:07:55.565279 4447 leader_election.cc:304] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3a520ce69d74074a3d309fd843bccd6, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.565377 4987 raft_consensus.cc:2798] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:55.565419 4987 raft_consensus.cc:695] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Becoming Leader. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Running, Role: LEADER
I20250110 03:07:55.565471 4987 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.566059 4239 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:55.674530 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:55.674700 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:55.675926 4999 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20250110 03:07:55.697134 4980 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:55.699077 4382 raft_consensus.cc:3054] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.700258 4980 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:55.707463 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:55.707501 4646 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:55.707783 4382 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:55.707990 4996 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:55.708357 4996 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:55.708474 4987 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:55.710616 5024 mvcc.cc:204] Tried to move back new op lower bound from 7112615836495921152 to 7112615835916410880. Current Snapshot: MvccSnapshot[applied={T|T < 7112615836495921152}]
I20250110 03:07:55.711903 4645 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:55.712293 5034 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:55.712651 5027 mvcc.cc:204] Tried to move back new op lower bound from 7112615836495921152 to 7112615835916410880. Current Snapshot: MvccSnapshot[applied={T|T < 7112615836495921152}]
I20250110 03:07:55.714179 5023 mvcc.cc:204] Tried to move back new op lower bound from 7112615836515987456 to 7112615835504852992. Current Snapshot: MvccSnapshot[applied={T|T < 7112615836495622144}]
INSERT report
rows total: 2000
time total: 23.5495 ms
time per row: 0.0117748 ms
I20250110 03:07:55.723438 4240 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51554:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250110 03:07:55.723575 4240 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 03:07:55.724850 4361 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.724867 4362 tablet_service.cc:1467] Processing CreateTablet for tablet ef7d7b7f5c444da0bfba41db5a231c30 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.724905 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.724910 4492 tablet_service.cc:1467] Processing CreateTablet for tablet ef7d7b7f5c444da0bfba41db5a231c30 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.724998 4362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7d7b7f5c444da0bfba41db5a231c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.725035 4492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7d7b7f5c444da0bfba41db5a231c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.725085 4360 tablet_service.cc:1467] Processing CreateTablet for tablet a99a80262a3d492d838cc0e0fc757ffc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.725104 4361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.725154 4360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a99a80262a3d492d838cc0e0fc757ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.725188 4494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.726315 4975 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:55.726372 4493 tablet_service.cc:1467] Processing CreateTablet for tablet a99a80262a3d492d838cc0e0fc757ffc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.726482 4493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a99a80262a3d492d838cc0e0fc757ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.726662 4977 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:55.727057 4977 tablet_bootstrap.cc:654] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.727504 4975 tablet_bootstrap.cc:654] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.728003 4626 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.728013 4624 tablet_service.cc:1467] Processing CreateTablet for tablet a99a80262a3d492d838cc0e0fc757ffc (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.728138 4626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.728143 4977 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:55.728181 4977 ts_tablet_manager.cc:1397] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250110 03:07:55.728236 4624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a99a80262a3d492d838cc0e0fc757ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.728302 4977 raft_consensus.cc:357] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.728343 4977 raft_consensus.cc:383] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.728356 4977 raft_consensus.cc:738] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.728394 4977 consensus_queue.cc:260] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.728482 4977 ts_tablet_manager.cc:1428] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.728524 4977 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:55.728838 4977 tablet_bootstrap.cc:654] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.729278 4976 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:55.729293 4625 tablet_service.cc:1467] Processing CreateTablet for tablet ef7d7b7f5c444da0bfba41db5a231c30 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:55.729440 4625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7d7b7f5c444da0bfba41db5a231c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:55.729575 4975 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:55.729635 4975 ts_tablet_manager.cc:1397] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:55.729692 4976 tablet_bootstrap.cc:654] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.729785 4975 raft_consensus.cc:357] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.729875 4975 raft_consensus.cc:383] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.729902 4975 raft_consensus.cc:738] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.729949 4975 consensus_queue.cc:260] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.730050 4975 ts_tablet_manager.cc:1428] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:55.730106 4975 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:55.730401 4976 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:55.730454 4976 ts_tablet_manager.cc:1397] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.730497 4975 tablet_bootstrap.cc:654] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.730620 4976 raft_consensus.cc:357] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.730691 4976 raft_consensus.cc:383] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.730716 4976 raft_consensus.cc:738] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.730757 4976 consensus_queue.cc:260] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.730872 4976 ts_tablet_manager.cc:1428] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:55.730940 4976 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:55.731081 4977 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:55.731128 4977 ts_tablet_manager.cc:1397] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:55.731283 4977 raft_consensus.cc:357] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.731353 4977 raft_consensus.cc:383] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.731375 4977 raft_consensus.cc:738] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.731426 4977 consensus_queue.cc:260] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.731514 4977 ts_tablet_manager.cc:1428] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.731567 4976 tablet_bootstrap.cc:654] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.731577 4977 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:55.732183 4977 tablet_bootstrap.cc:654] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.732223 4976 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:55.732265 4976 ts_tablet_manager.cc:1397] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.732429 4976 raft_consensus.cc:357] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.732504 4976 raft_consensus.cc:383] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.732529 4976 raft_consensus.cc:738] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.732570 4976 consensus_queue.cc:260] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.732841 4977 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:55.732846 4976 ts_tablet_manager.cc:1428] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.732883 4977 ts_tablet_manager.cc:1397] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.732908 4976 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:55.733042 4977 raft_consensus.cc:357] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.733124 4977 raft_consensus.cc:383] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.733148 4977 raft_consensus.cc:738] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.733160 4975 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:55.733197 4975 ts_tablet_manager.cc:1397] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:55.733191 4977 consensus_queue.cc:260] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.733283 4977 ts_tablet_manager.cc:1428] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.733315 4975 raft_consensus.cc:357] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.733371 4976 tablet_bootstrap.cc:654] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.733374 4975 raft_consensus.cc:383] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.733394 4975 raft_consensus.cc:738] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.733434 4975 consensus_queue.cc:260] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.733537 4975 ts_tablet_manager.cc:1428] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.733603 4975 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:55.733958 4976 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:55.734006 4976 ts_tablet_manager.cc:1397] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.734169 4975 tablet_bootstrap.cc:654] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:55.734148 4976 raft_consensus.cc:357] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.734203 4976 raft_consensus.cc:383] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.734221 4976 raft_consensus.cc:738] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.734290 4976 consensus_queue.cc:260] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.734418 4976 ts_tablet_manager.cc:1428] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.734867 4975 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:55.734907 4975 ts_tablet_manager.cc:1397] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:55.735025 4975 raft_consensus.cc:357] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.735069 4975 raft_consensus.cc:383] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:55.735090 4975 raft_consensus.cc:738] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:55.735136 4975 consensus_queue.cc:260] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.735226 4975 ts_tablet_manager.cc:1428] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:55.856915 5035 raft_consensus.cc:491] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:55.856997 5035 raft_consensus.cc:513] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.857136 5035 leader_election.cc:290] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:55.857309 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7d7b7f5c444da0bfba41db5a231c30" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" is_pre_election: true
I20250110 03:07:55.857414 4514 raft_consensus.cc:2463] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:55.857378 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7d7b7f5c444da0bfba41db5a231c30" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:55.857475 4381 raft_consensus.cc:2463] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:55.857581 4579 leader_election.cc:304] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.857672 5035 raft_consensus.cc:2798] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:55.857712 5035 raft_consensus.cc:491] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:55.857731 5035 raft_consensus.cc:3054] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.858309 5035 raft_consensus.cc:513] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.858405 5035 leader_election.cc:290] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Requested vote from peers e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:55.858580 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7d7b7f5c444da0bfba41db5a231c30" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:07:55.858647 4514 raft_consensus.cc:3054] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.858618 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7d7b7f5c444da0bfba41db5a231c30" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:55.858705 4381 raft_consensus.cc:3054] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.859268 4514 raft_consensus.cc:2463] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:55.859422 4579 leader_election.cc:304] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:55.859447 4381 raft_consensus.cc:2463] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:55.859501 5035 raft_consensus.cc:2798] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:55.859558 5035 raft_consensus.cc:695] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Becoming Leader. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Running, Role: LEADER
I20250110 03:07:55.859613 5035 consensus_queue.cc:237] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.860141 4240 catalog_manager.cc:5526] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 reported cstate change: term changed from 0 to 1, leader changed from <none> to b680ae46493d4601a57d0545c65a1895 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:55.908429 5035 raft_consensus.cc:491] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:55.908521 5035 raft_consensus.cc:513] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.908717 5035 leader_election.cc:290] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:55.908954 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99a80262a3d492d838cc0e0fc757ffc" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:55.909030 4381 raft_consensus.cc:2463] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:55.908984 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99a80262a3d492d838cc0e0fc757ffc" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" is_pre_election: true
I20250110 03:07:55.909077 4514 raft_consensus.cc:2463] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:55.909191 4579 leader_election.cc:304] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e3a520ce69d74074a3d309fd843bccd6; no voters:
I20250110 03:07:55.909296 5035 raft_consensus.cc:2798] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:55.909332 5035 raft_consensus.cc:491] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:55.909358 5035 raft_consensus.cc:3054] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.909938 5035 raft_consensus.cc:513] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.910064 5035 leader_election.cc:290] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Requested vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:55.910202 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99a80262a3d492d838cc0e0fc757ffc" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:55.910269 4381 raft_consensus.cc:3054] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.910230 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a99a80262a3d492d838cc0e0fc757ffc" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:07:55.910296 4514 raft_consensus.cc:3054] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:55.910821 4381 raft_consensus.cc:2463] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:55.910873 4514 raft_consensus.cc:2463] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:55.910980 4579 leader_election.cc:304] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e3a520ce69d74074a3d309fd843bccd6; no voters:
I20250110 03:07:55.911082 5035 raft_consensus.cc:2798] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:55.911152 5035 raft_consensus.cc:695] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Becoming Leader. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Running, Role: LEADER
I20250110 03:07:55.911212 5035 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:55.911784 4240 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 reported cstate change: term changed from 0 to 1, leader changed from <none> to b680ae46493d4601a57d0545c65a1895 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:56.050055 4982 raft_consensus.cc:491] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:56.050141 4982 raft_consensus.cc:513] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.050406 4982 leader_election.cc:290] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:56.052937 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "e3a520ce69d74074a3d309fd843bccd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895" is_pre_election: true
I20250110 03:07:56.052979 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "e3a520ce69d74074a3d309fd843bccd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" is_pre_election: true
I20250110 03:07:56.053030 4646 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a520ce69d74074a3d309fd843bccd6 in term 0.
I20250110 03:07:56.053045 4514 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3a520ce69d74074a3d309fd843bccd6 in term 0.
I20250110 03:07:56.053215 4315 leader_election.cc:304] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3a520ce69d74074a3d309fd843bccd6, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:56.053340 4982 raft_consensus.cc:2798] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:56.053383 4982 raft_consensus.cc:491] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:56.053403 4982 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.053984 4982 raft_consensus.cc:513] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.054091 4982 leader_election.cc:290] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [CANDIDATE]: Term 1 election: Requested vote from peers e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:56.054229 4514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "e3a520ce69d74074a3d309fd843bccd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:07:56.054256 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "e3a520ce69d74074a3d309fd843bccd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895"
I20250110 03:07:56.054289 4514 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.054304 4646 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.054791 4514 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a520ce69d74074a3d309fd843bccd6 in term 1.
I20250110 03:07:56.054836 4646 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3a520ce69d74074a3d309fd843bccd6 in term 1.
I20250110 03:07:56.054944 4315 leader_election.cc:304] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3a520ce69d74074a3d309fd843bccd6, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:56.055027 4982 raft_consensus.cc:2798] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:56.055099 4982 raft_consensus.cc:695] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Becoming Leader. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Running, Role: LEADER
I20250110 03:07:56.055250 4982 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.056087 4240 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129). New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:56.154361 5067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:56.154500 5067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:56.155710 5067 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20250110 03:07:56.187271 4381 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.187326 4513 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.187546 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.187533 5035 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.187770 5035 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.187906 4514 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.188156 4514 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.188445 4982 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.188864 4646 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.188870 4992 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 03:07:56.188884 5035 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 03:07:56.189627 4982 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 29.6641 ms
time per row: 0.014832 ms
I20250110 03:07:56.209102 4240 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51554:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20250110 03:07:56.209251 4240 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 03:07:56.210569 4362 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6206c7b3014154ba2689f574532355 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.210611 4361 tablet_service.cc:1467] Processing CreateTablet for tablet ec019c792274403585d757840bdf84aa (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.210697 4362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6206c7b3014154ba2689f574532355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.210837 4494 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6206c7b3014154ba2689f574532355 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.210837 4493 tablet_service.cc:1467] Processing CreateTablet for tablet ec019c792274403585d757840bdf84aa (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.210884 4492 tablet_service.cc:1467] Processing CreateTablet for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.210997 4494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6206c7b3014154ba2689f574532355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.211146 4626 tablet_service.cc:1467] Processing CreateTablet for tablet ec019c792274403585d757840bdf84aa (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.211170 4625 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6206c7b3014154ba2689f574532355 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.211252 4626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec019c792274403585d757840bdf84aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.211275 4624 tablet_service.cc:1467] Processing CreateTablet for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.211364 4624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.211979 4360 tablet_service.cc:1467] Processing CreateTablet for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 (DEFAULT_TABLE table=rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250110 03:07:56.212073 4360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.212316 4975 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:56.212551 4976 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:56.212687 4975 tablet_bootstrap.cc:654] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.212967 4976 tablet_bootstrap.cc:654] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.213119 4977 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:56.213265 4493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec019c792274403585d757840bdf84aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.213464 4977 tablet_bootstrap.cc:654] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.213554 4976 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:56.213601 4976 ts_tablet_manager.cc:1397] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:56.213730 4976 raft_consensus.cc:357] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.213810 4976 raft_consensus.cc:383] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.213837 4976 raft_consensus.cc:738] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.213887 4976 consensus_queue.cc:260] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.214010 4976 ts_tablet_manager.cc:1428] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.214058 4976 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:56.214151 4977 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:56.214182 4977 ts_tablet_manager.cc:1397] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:56.214275 4977 raft_consensus.cc:357] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.214310 4977 raft_consensus.cc:383] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.214323 4977 raft_consensus.cc:738] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.214351 4977 consensus_queue.cc:260] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.214403 4977 ts_tablet_manager.cc:1428] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.214428 4976 tablet_bootstrap.cc:654] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.214438 4492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.214450 4977 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:56.214824 4977 tablet_bootstrap.cc:654] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.215091 4976 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:56.215129 4976 ts_tablet_manager.cc:1397] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:56.215257 4976 raft_consensus.cc:357] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.215319 4976 raft_consensus.cc:383] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.215351 4976 raft_consensus.cc:738] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.215385 4977 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:56.215394 4976 consensus_queue.cc:260] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.215436 4977 ts_tablet_manager.cc:1397] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:56.215471 4976 ts_tablet_manager.cc:1428] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.215574 4977 raft_consensus.cc:357] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.215638 4977 raft_consensus.cc:383] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.215655 4625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6206c7b3014154ba2689f574532355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.215667 4977 raft_consensus.cc:738] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.215704 4977 consensus_queue.cc:260] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.215835 4977 ts_tablet_manager.cc:1428] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20250110 03:07:56.216612 4361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec019c792274403585d757840bdf84aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:56.216607 4976 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: Bootstrap starting.
I20250110 03:07:56.216995 4976 tablet_bootstrap.cc:654] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.217487 4976 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: No bootstrap required, opened a new log
I20250110 03:07:56.217518 4977 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: Bootstrap starting.
I20250110 03:07:56.217528 4976 ts_tablet_manager.cc:1397] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:56.217653 4976 raft_consensus.cc:357] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.217710 4976 raft_consensus.cc:383] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.217729 4976 raft_consensus.cc:738] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.217765 4976 consensus_queue.cc:260] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.217842 4976 ts_tablet_manager.cc:1428] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.217864 4977 tablet_bootstrap.cc:654] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.217998 4975 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:56.218040 4975 ts_tablet_manager.cc:1397] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20250110 03:07:56.218166 4975 raft_consensus.cc:357] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.218231 4975 raft_consensus.cc:383] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.218267 4975 raft_consensus.cc:738] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.218310 4975 consensus_queue.cc:260] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.218395 4975 ts_tablet_manager.cc:1428] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.218405 4977 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: No bootstrap required, opened a new log
I20250110 03:07:56.218441 4977 ts_tablet_manager.cc:1397] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20250110 03:07:56.218444 4975 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:56.218531 4977 raft_consensus.cc:357] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.218703 4977 raft_consensus.cc:383] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.218732 4977 raft_consensus.cc:738] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.218735 4975 tablet_bootstrap.cc:654] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.218770 4977 consensus_queue.cc:260] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.218843 4977 ts_tablet_manager.cc:1428] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.219283 4975 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:56.219336 4975 ts_tablet_manager.cc:1397] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:56.219461 4975 raft_consensus.cc:357] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.219507 4975 raft_consensus.cc:383] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.219521 4975 raft_consensus.cc:738] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.219566 4975 consensus_queue.cc:260] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.219651 4975 ts_tablet_manager.cc:1428] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.219704 4975 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:07:56.220218 4975 tablet_bootstrap.cc:654] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: Neither blocks nor log segments found. Creating new log.
I20250110 03:07:56.220659 4975 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: No bootstrap required, opened a new log
I20250110 03:07:56.220692 4975 ts_tablet_manager.cc:1397] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:56.220781 4975 raft_consensus.cc:357] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.220827 4975 raft_consensus.cc:383] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 03:07:56.220840 4975 raft_consensus.cc:738] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: FOLLOWER
I20250110 03:07:56.220870 4975 consensus_queue.cc:260] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.220956 4975 ts_tablet_manager.cc:1428] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.249789 4996 raft_consensus.cc:491] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:56.249847 4996 raft_consensus.cc:513] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.249982 4996 leader_election.cc:290] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:56.250124 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cecae3ae274ba9bcdf8df3a8ebdf46" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:56.250190 4382 raft_consensus.cc:2463] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 0.
I20250110 03:07:56.250164 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cecae3ae274ba9bcdf8df3a8ebdf46" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895" is_pre_election: true
I20250110 03:07:56.250224 4646 raft_consensus.cc:2463] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 0.
I20250110 03:07:56.250310 4447 leader_election.cc:304] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3a520ce69d74074a3d309fd843bccd6, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:56.250388 4996 raft_consensus.cc:2798] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:56.250422 4996 raft_consensus.cc:491] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:56.250442 4996 raft_consensus.cc:3054] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.250994 4996 raft_consensus.cc:513] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.251113 4996 leader_election.cc:290] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 election: Requested vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:56.251232 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cecae3ae274ba9bcdf8df3a8ebdf46" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b680ae46493d4601a57d0545c65a1895"
I20250110 03:07:56.251241 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cecae3ae274ba9bcdf8df3a8ebdf46" candidate_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:56.251298 4646 raft_consensus.cc:3054] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.251300 4382 raft_consensus.cc:3054] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.251765 4382 raft_consensus.cc:2463] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 1.
I20250110 03:07:56.251767 4646 raft_consensus.cc:2463] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ba4f91a2c94c3883d60f7002e12bb2 in term 1.
I20250110 03:07:56.251922 4448 leader_election.cc:304] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:56.252018 4996 raft_consensus.cc:2798] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:56.252130 4996 raft_consensus.cc:695] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Becoming Leader. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Running, Role: LEADER
I20250110 03:07:56.252183 4996 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.252624 4240 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130). New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:56.315311 5102 raft_consensus.cc:491] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:56.315399 5102 raft_consensus.cc:513] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.315541 5102 leader_election.cc:290] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:56.315719 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6206c7b3014154ba2689f574532355" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:56.315757 4513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6206c7b3014154ba2689f574532355" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" is_pre_election: true
I20250110 03:07:56.315806 4382 raft_consensus.cc:2463] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:56.315840 4513 raft_consensus.cc:2463] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:56.315943 4579 leader_election.cc:304] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e3a520ce69d74074a3d309fd843bccd6; no voters:
I20250110 03:07:56.316047 5102 raft_consensus.cc:2798] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:56.316094 5102 raft_consensus.cc:491] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:56.316123 5102 raft_consensus.cc:3054] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.316918 5102 raft_consensus.cc:513] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.317029 5102 leader_election.cc:290] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Requested vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:56.317183 4513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6206c7b3014154ba2689f574532355" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:07:56.317180 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6206c7b3014154ba2689f574532355" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:56.317250 4382 raft_consensus.cc:3054] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.317250 4513 raft_consensus.cc:3054] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.317902 4382 raft_consensus.cc:2463] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:56.317983 4513 raft_consensus.cc:2463] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:56.318063 4579 leader_election.cc:304] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e3a520ce69d74074a3d309fd843bccd6; no voters:
I20250110 03:07:56.318148 5102 raft_consensus.cc:2798] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:56.318193 5102 raft_consensus.cc:695] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Becoming Leader. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Running, Role: LEADER
I20250110 03:07:56.318240 5102 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.318706 4240 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 reported cstate change: term changed from 0 to 1, leader changed from <none> to b680ae46493d4601a57d0545c65a1895 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } }
I20250110 03:07:56.358613 5102 raft_consensus.cc:491] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 03:07:56.358671 5102 raft_consensus.cc:513] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.358762 5102 leader_election.cc:290] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:56.358932 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec019c792274403585d757840bdf84aa" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6" is_pre_election: true
I20250110 03:07:56.358937 4513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec019c792274403585d757840bdf84aa" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" is_pre_election: true
I20250110 03:07:56.358999 4513 raft_consensus.cc:2463] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:56.359000 4382 raft_consensus.cc:2463] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 0.
I20250110 03:07:56.359110 4579 leader_election.cc:304] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e3a520ce69d74074a3d309fd843bccd6; no voters:
I20250110 03:07:56.359201 5102 raft_consensus.cc:2798] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 03:07:56.359244 5102 raft_consensus.cc:491] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 03:07:56.359261 5102 raft_consensus.cc:3054] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.359730 5102 raft_consensus.cc:513] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.359848 5102 leader_election.cc:290] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Requested vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:56.360006 4382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec019c792274403585d757840bdf84aa" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:07:56.360028 4513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec019c792274403585d757840bdf84aa" candidate_uuid: "b680ae46493d4601a57d0545c65a1895" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:07:56.360064 4382 raft_consensus.cc:3054] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.360085 4513 raft_consensus.cc:3054] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 03:07:56.360531 4513 raft_consensus.cc:2463] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:56.360545 4382 raft_consensus.cc:2463] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b680ae46493d4601a57d0545c65a1895 in term 1.
I20250110 03:07:56.360697 4579 leader_election.cc:304] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b680ae46493d4601a57d0545c65a1895, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:07:56.360797 5102 raft_consensus.cc:2798] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 03:07:56.360855 5102 raft_consensus.cc:695] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Becoming Leader. State: Replica: b680ae46493d4601a57d0545c65a1895, State: Running, Role: LEADER
I20250110 03:07:56.360894 5102 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:07:56.361364 4240 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 reported cstate change: term changed from 0 to 1, leader changed from <none> to b680ae46493d4601a57d0545c65a1895 (127.14.19.131). New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } }
W20250110 03:07:56.472640 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:56.472808 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:56.474049 5116 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20250110 03:07:56.503756 4382 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.503751 4513 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.503953 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.504077 4646 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.504043 5102 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.504258 4996 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.504316 4980 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.504731 5112 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.505781 4382 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.506120 4512 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 03:07:56.506378 5035 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 03:07:56.506829 5035 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 21.821 ms
time per row: 0.0109105 ms
I20250110 03:07:56.518780 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.133:34025
--local_ip_for_outbound_sockets=127.14.19.133
--tserver_master_addrs=127.14.19.190:39657
--webserver_port=39041
--webserver_interface=127.14.19.133
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250110 03:07:56.592839 5153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:56.592984 5153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:56.593003 5153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:56.594314 5153 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:56.594369 5153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:56.594408 5153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.133
I20250110 03:07:56.596570 5153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.133:34025
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.14.19.133
--webserver_port=39041
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.5153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.133
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:56.596805 5153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:56.597075 5153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:56.603192 5161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:56.604027 5158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:56.605115 5153 server_base.cc:1034] running on GCE node
W20250110 03:07:56.605314 5159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:56.605502 5153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:56.605684 5153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:56.606809 5153 hybrid_clock.cc:648] HybridClock initialized: now 1736478476606793 us; error 30 us; skew 500 ppm
I20250110 03:07:56.607977 5153 webserver.cc:458] Webserver started at http://127.14.19.133:39041/ using document root <none> and password file <none>
I20250110 03:07:56.608141 5153 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:56.608194 5153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:56.609370 5153 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:56.609889 5167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.610045 5153 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250110 03:07:56.610112 5153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8cb2e4afb3384e40879d605ea471b039"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:56.610380 5153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:56.623677 5153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:56.623875 5153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:56.623958 5153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:56.624119 5153 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:56.624394 5153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:56.624423 5153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.624444 5153 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:56.624468 5153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.629954 5153 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.133:34025
I20250110 03:07:56.630012 5280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.133:34025 every 8 connection(s)
I20250110 03:07:56.630283 5153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 03:07:56.632642 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 5153
I20250110 03:07:56.632891 14414 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskWQrDfu/build/release/bin/kudu
/tmp/dist-test-taskWQrDfu/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.19.132:41431
--local_ip_for_outbound_sockets=127.14.19.132
--tserver_master_addrs=127.14.19.190:39657
--webserver_port=41275
--webserver_interface=127.14.19.132
--builtin_ntp_servers=127.14.19.148:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250110 03:07:56.634606 5281 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:56.634703 5281 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:56.634891 5281 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:56.635277 4240 ts_manager.cc:194] Re-registered known tserver with Master: 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025)
I20250110 03:07:56.635771 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.133:40525
W20250110 03:07:56.708582 5284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 03:07:56.708719 5284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 03:07:56.708736 5284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 03:07:56.709960 5284 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250110 03:07:56.710001 5284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 03:07:56.710024 5284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.19.132
I20250110 03:07:56.711413 5284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.19.148:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.19.132:41431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.14.19.132
--webserver_port=41275
--tserver_master_addrs=127.14.19.190:39657
--never_fsync=true
--heap_profile_path=/tmp/kudu.5284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.19.132
--log_dir=/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type RELEASE
built by None at 10 Jan 2025 02:43:12 UTC on a633016ff5c9
build id 4102
I20250110 03:07:56.711606 5284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 03:07:56.711817 5284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 03:07:56.716549 5292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:56.716702 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 03:07:56.717617 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 03:07:56.718403 5284 server_base.cc:1034] running on GCE node
I20250110 03:07:56.718590 5284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 03:07:56.718777 5284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 03:07:56.719902 5284 hybrid_clock.cc:648] HybridClock initialized: now 1736478476719884 us; error 33 us; skew 500 ppm
I20250110 03:07:56.721015 5284 webserver.cc:458] Webserver started at http://127.14.19.132:41275/ using document root <none> and password file <none>
I20250110 03:07:56.721190 5284 fs_manager.cc:362] Metadata directory not provided
I20250110 03:07:56.721247 5284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 03:07:56.722369 5284 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250110 03:07:56.722891 5298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.723042 5284 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.723110 5284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "22901baa49564439bfd6978fe2e4a4a9"
format_stamp: "Formatted at 2025-01-10 03:07:51 on dist-test-slave-nrvn"
I20250110 03:07:56.723367 5284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 03:07:56.737226 5284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 03:07:56.737401 5284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 03:07:56.737496 5284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 03:07:56.737648 5284 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 03:07:56.737890 5284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 03:07:56.737920 5284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.737951 5284 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 03:07:56.737980 5284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:56.743244 5284 rpc_server.cc:307] RPC server started. Bound to: 127.14.19.132:41431
I20250110 03:07:56.743302 5411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.14.19.132:41431 every 8 connection(s)
I20250110 03:07:56.743574 5284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 03:07:56.747785 14414 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskWQrDfu/build/release/bin/kudu as pid 5284
I20250110 03:07:56.749389 5412 heartbeater.cc:346] Connected to a master server at 127.14.19.190:39657
I20250110 03:07:56.749483 5412 heartbeater.cc:463] Registering TS with master...
I20250110 03:07:56.749658 5412 heartbeater.cc:510] Master 127.14.19.190:39657 requested a full tablet report, sending...
I20250110 03:07:56.750032 4240 ts_manager.cc:194] Re-registered known tserver with Master: 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431)
I20250110 03:07:56.750389 4240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.19.132:54995
I20250110 03:07:56.857321 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:56.858647 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:56.862534 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:56.862947 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:56.866186 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:56.904623 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:56.905625 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:56.908018 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:56.909026 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:56.912262 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:56.966215 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:56.966882 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:56.967233 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:56.971418 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:56.971657 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:57.005988 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:57.006209 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:57.007777 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:57.008317 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:57.010933 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:57.022259 4381 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.022866 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.023109 4646 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.023111 5100 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.023334 5100 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.023954 5557 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.024107 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.025228 4646 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.025370 4236 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250110 03:07:57.026369 4316 consensus_peers.cc:487] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cd35b6ae5b54b05bb6d93b79442ba9d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:57.030303 4646 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.030936 4381 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.031060 4513 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.031289 5569 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.031410 5102 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.032001 5102 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.032089 4513 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.032948 4233 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.033430 4381 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
W20250110 03:07:57.034952 4579 consensus_peers.cc:487] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8953c1791507457e805dd8a89638dd29. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:57.037371 4513 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.038012 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.038017 4646 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.038211 5578 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.038312 5579 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.038862 5578 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.038990 4646 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.039991 4240 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.040041 4381 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
W20250110 03:07:57.040930 4448 consensus_peers.cc:487] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5cecae3ae274ba9bcdf8df3a8ebdf46. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:57.047384 4646 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.047806 4513 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.047837 4381 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:57.047868 4579 consensus_peers.cc:487] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec019c792274403585d757840bdf84aa. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:57.047974 5586 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.048084 5569 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.048645 5569 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.048769 4513 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.048800 4381 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.049672 4240 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:57.053406 4513 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.053800 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.053817 4646 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:57.053927 4448 consensus_peers.cc:487] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56330dd8a2ef4597be2d809c1cf02000. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:57.053957 5152 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.054045 5578 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.054721 5579 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.054867 4646 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.055758 4233 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:57.056267 4381 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:57.059806 4646 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.060205 4513 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:07:57.060211 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250110 03:07:57.060315 4579 consensus_peers.cc:487] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a99a80262a3d492d838cc0e0fc757ffc. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:57.060420 5586 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.060501 5569 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:07:57.061057 5586 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.061218 4381 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.061244 4513 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:57.062106 4233 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index -1 to 4, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:57.562362 5690 ts_tablet_manager.cc:927] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:57.562585 5690 tablet_copy_client.cc:323] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:07:57.565111 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet a99a80262a3d492d838cc0e0fc757ffc from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39641)
I20250110 03:07:57.565191 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet a99a80262a3d492d838cc0e0fc757ffc from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39641: session id = 8cb2e4afb3384e40879d605ea471b039-a99a80262a3d492d838cc0e0fc757ffc
I20250110 03:07:57.565726 4666 tablet_copy_source_session.cc:215] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:57.565991 5693 ts_tablet_manager.cc:927] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:57.566150 5693 tablet_copy_client.cc:323] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:43399
I20250110 03:07:57.566520 5690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a99a80262a3d492d838cc0e0fc757ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:57.568684 4534 tablet_copy_service.cc:140] P e8ba4f91a2c94c3883d60f7002e12bb2: Received BeginTabletCopySession request for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:52665)
I20250110 03:07:57.568756 4534 tablet_copy_service.cc:161] P e8ba4f91a2c94c3883d60f7002e12bb2: Beginning new tablet copy session on tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:52665: session id = 22901baa49564439bfd6978fe2e4a4a9-e5cecae3ae274ba9bcdf8df3a8ebdf46
I20250110 03:07:57.568869 5690 tablet_copy_client.cc:806] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:57.569002 5690 tablet_copy_client.cc:670] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:57.569393 4534 tablet_copy_source_session.cc:215] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:57.569764 5690 tablet_copy_client.cc:538] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:57.569939 5696 ts_tablet_manager.cc:927] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:07:57.570102 5696 tablet_copy_client.cc:323] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:43399
I20250110 03:07:57.570128 5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:57.570278 4534 tablet_copy_service.cc:140] P e8ba4f91a2c94c3883d60f7002e12bb2: Received BeginTabletCopySession request for tablet 56330dd8a2ef4597be2d809c1cf02000 from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:52665)
I20250110 03:07:57.570328 4534 tablet_copy_service.cc:161] P e8ba4f91a2c94c3883d60f7002e12bb2: Beginning new tablet copy session on tablet 56330dd8a2ef4597be2d809c1cf02000 from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:52665: session id = 22901baa49564439bfd6978fe2e4a4a9-56330dd8a2ef4597be2d809c1cf02000
I20250110 03:07:57.570528 5690 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:07:57.570789 4534 tablet_copy_source_session.cc:215] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:57.571202 5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56330dd8a2ef4597be2d809c1cf02000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:57.572273 5690 log.cc:826] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:57.573282 5696 tablet_copy_client.cc:806] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:57.573362 5693 tablet_copy_client.cc:806] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:57.573415 5696 tablet_copy_client.cc:670] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:57.573487 5693 tablet_copy_client.cc:670] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:57.574128 5693 tablet_copy_client.cc:538] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:57.574609 5696 tablet_copy_client.cc:538] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:57.574716 5693 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:07:57.575210 5696 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:07:57.575994 5690 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:57.576210 5690 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:07:57.576313 5690 ts_tablet_manager.cc:1397] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250110 03:07:57.576484 5693 log.cc:826] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Log is configured to *not* fsync() on all Append() calls
I20250110 03:07:57.577215 5690 raft_consensus.cc:357] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.577329 5690 raft_consensus.cc:738] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:07:57.577390 5690 consensus_queue.cc:260] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.577598 5690 ts_tablet_manager.cc:1428] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250110 03:07:57.577703 5281 heartbeater.cc:502] Master 127.14.19.190:39657 was elected leader, sending a full tablet report...
I20250110 03:07:57.577883 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-a99a80262a3d492d838cc0e0fc757ffc received from {username='slave'} at 127.14.19.133:39641
I20250110 03:07:57.577937 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-a99a80262a3d492d838cc0e0fc757ffc on tablet a99a80262a3d492d838cc0e0fc757ffc with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:07:57.580669 5696 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:57.580669 5693 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:57.580925 5696 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:07:57.580931 5693 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:07:57.581043 5696 ts_tablet_manager.cc:1397] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250110 03:07:57.581043 5693 ts_tablet_manager.cc:1397] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20250110 03:07:57.581847 5693 raft_consensus.cc:357] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.581847 5696 raft_consensus.cc:357] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.582032 5693 raft_consensus.cc:738] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:07:57.582038 5696 raft_consensus.cc:738] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:07:57.582110 5693 consensus_queue.cc:260] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.582110 5696 consensus_queue.cc:260] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.582360 5696 ts_tablet_manager.cc:1428] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250110 03:07:57.582360 5693 ts_tablet_manager.cc:1428] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250110 03:07:57.582402 5412 heartbeater.cc:502] Master 127.14.19.190:39657 was elected leader, sending a full tablet report...
I20250110 03:07:57.582667 4533 tablet_copy_service.cc:342] P e8ba4f91a2c94c3883d60f7002e12bb2: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-56330dd8a2ef4597be2d809c1cf02000 received from {username='slave'} at 127.14.19.132:52665
I20250110 03:07:57.582724 4533 tablet_copy_service.cc:434] P e8ba4f91a2c94c3883d60f7002e12bb2: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-56330dd8a2ef4597be2d809c1cf02000 on tablet 56330dd8a2ef4597be2d809c1cf02000 with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:07:57.582850 4534 tablet_copy_service.cc:342] P e8ba4f91a2c94c3883d60f7002e12bb2: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-e5cecae3ae274ba9bcdf8df3a8ebdf46 received from {username='slave'} at 127.14.19.132:52665
I20250110 03:07:57.582906 4534 tablet_copy_service.cc:434] P e8ba4f91a2c94c3883d60f7002e12bb2: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-e5cecae3ae274ba9bcdf8df3a8ebdf46 on tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:07:57.612643 5690 ts_tablet_manager.cc:927] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:57.612774 5690 tablet_copy_client.cc:323] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:07:57.613005 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet ec019c792274403585d757840bdf84aa from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39641)
I20250110 03:07:57.613073 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet ec019c792274403585d757840bdf84aa from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39641: session id = 8cb2e4afb3384e40879d605ea471b039-ec019c792274403585d757840bdf84aa
I20250110 03:07:57.613453 4666 tablet_copy_source_session.cc:215] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:57.613830 5690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec019c792274403585d757840bdf84aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:57.614810 5690 tablet_copy_client.cc:806] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:57.614907 5690 tablet_copy_client.cc:670] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:57.615502 5690 tablet_copy_client.cc:538] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:57.616050 5690 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:07:57.619132 5690 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:57.619304 5690 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:07:57.619386 5690 ts_tablet_manager.cc:1397] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:07:57.619488 5690 raft_consensus.cc:357] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.619545 5690 raft_consensus.cc:738] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:07:57.619570 5690 consensus_queue.cc:260] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.619655 5690 ts_tablet_manager.cc:1428] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:57.619851 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-ec019c792274403585d757840bdf84aa received from {username='slave'} at 127.14.19.133:39641
I20250110 03:07:57.619908 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-ec019c792274403585d757840bdf84aa on tablet ec019c792274403585d757840bdf84aa with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:07:57.627941 5693 ts_tablet_manager.cc:927] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:57.628055 5693 tablet_copy_client.cc:323] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:37651
I20250110 03:07:57.630317 4402 tablet_copy_service.cc:140] P e3a520ce69d74074a3d309fd843bccd6: Received BeginTabletCopySession request for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:41109)
I20250110 03:07:57.630389 4402 tablet_copy_service.cc:161] P e3a520ce69d74074a3d309fd843bccd6: Beginning new tablet copy session on tablet 1cd35b6ae5b54b05bb6d93b79442ba9d from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:41109: session id = 22901baa49564439bfd6978fe2e4a4a9-1cd35b6ae5b54b05bb6d93b79442ba9d
I20250110 03:07:57.630928 4402 tablet_copy_source_session.cc:215] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:57.631363 5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:57.632258 5693 tablet_copy_client.cc:806] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:57.632347 5693 tablet_copy_client.cc:670] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:57.633008 5693 tablet_copy_client.cc:538] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:57.633513 5693 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:07:57.636296 5693 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:57.636482 5693 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:07:57.636564 5693 ts_tablet_manager.cc:1397] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250110 03:07:57.636680 5693 raft_consensus.cc:357] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.636734 5693 raft_consensus.cc:738] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:07:57.636761 5693 consensus_queue.cc:260] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:57.636849 5693 ts_tablet_manager.cc:1428] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:57.637080 4402 tablet_copy_service.cc:342] P e3a520ce69d74074a3d309fd843bccd6: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-1cd35b6ae5b54b05bb6d93b79442ba9d received from {username='slave'} at 127.14.19.132:41109
I20250110 03:07:57.637136 4402 tablet_copy_service.cc:434] P e3a520ce69d74074a3d309fd843bccd6: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-1cd35b6ae5b54b05bb6d93b79442ba9d on tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:07:57.650919 5690 ts_tablet_manager.cc:927] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:57.651041 5690 tablet_copy_client.cc:323] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:07:57.651218 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet 8953c1791507457e805dd8a89638dd29 from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39641)
I20250110 03:07:57.651268 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet 8953c1791507457e805dd8a89638dd29 from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39641: session id = 8cb2e4afb3384e40879d605ea471b039-8953c1791507457e805dd8a89638dd29
I20250110 03:07:57.651577 4666 tablet_copy_source_session.cc:215] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:57.651880 5690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8953c1791507457e805dd8a89638dd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:57.652763 5690 tablet_copy_client.cc:806] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:57.652886 5690 tablet_copy_client.cc:670] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:57.653550 5690 tablet_copy_client.cc:538] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:57.654013 5690 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:07:57.656251 5690 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:57.656431 5690 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:07:57.656517 5690 ts_tablet_manager.cc:1397] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20250110 03:07:57.656630 5690 raft_consensus.cc:357] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.656682 5690 raft_consensus.cc:738] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:07:57.656713 5690 consensus_queue.cc:260] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:57.656801 5690 ts_tablet_manager.cc:1428] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:57.656981 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-8953c1791507457e805dd8a89638dd29 received from {username='slave'} at 127.14.19.133:39641
I20250110 03:07:57.657028 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-8953c1791507457e805dd8a89638dd29 on tablet 8953c1791507457e805dd8a89638dd29 with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:07:58.024273 5366 raft_consensus.cc:1212] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:58.027321 5235 raft_consensus.cc:1212] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:58.111311 5366 raft_consensus.cc:1212] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:58.187563 5366 raft_consensus.cc:1212] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:58.206156 5235 raft_consensus.cc:1212] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:58.269997 5235 raft_consensus.cc:1212] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250110 03:07:58.458873 5569 raft_consensus.cc:1059] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:07:58.459079 5569 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:58.459617 5235 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.459616 4513 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.459632 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.459890 5836 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.460033 5569 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.460172 5771 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.460963 5804 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.461129 4513 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.461138 5235 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.462313 4236 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.462390 4381 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.463655 4646 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:58.464012 5235 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.464031 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.464185 5569 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.464267 5771 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.464769 5804 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.464915 4381 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.465713 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a99a80262a3d492d838cc0e0fc757ffc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:58.465900 4233 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.466171 5235 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.467216 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet a99a80262a3d492d838cc0e0fc757ffc with delete_type TABLET_DATA_TOMBSTONED (TS e8ba4f91a2c94c3883d60f7002e12bb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51178
I20250110 03:07:58.467417 5844 tablet_replica.cc:331] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: stopping tablet replica
I20250110 03:07:58.467547 5844 raft_consensus.cc:2238] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:58.467628 5844 raft_consensus.cc:2267] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:58.467909 5844 ts_tablet_manager.cc:1905] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:58.469134 5844 ts_tablet_manager.cc:1918] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:58.469219 5844 log.cc:1198] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/a99a80262a3d492d838cc0e0fc757ffc
I20250110 03:07:58.469599 4224 catalog_manager.cc:4872] TS e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399): tablet a99a80262a3d492d838cc0e0fc757ffc (table rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]) successfully deleted
I20250110 03:07:58.589468 5770 raft_consensus.cc:1059] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:07:58.589684 5770 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:58.590210 4646 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.590224 5366 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.590214 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.590438 5579 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.590584 5846 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.590714 5770 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.591768 5770 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.591902 4381 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.592931 4233 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.592928 5366 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.593379 4646 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.594688 4513 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:58.595021 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.595068 5366 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.595237 5846 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.595325 5579 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.595925 5579 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.596104 4381 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.596735 4224 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 56330dd8a2ef4597be2d809c1cf02000 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:58.596881 5366 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.596930 4236 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:58.597728 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet 56330dd8a2ef4597be2d809c1cf02000 with delete_type TABLET_DATA_TOMBSTONED (TS b680ae46493d4601a57d0545c65a1895 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57156
I20250110 03:07:58.598033 5850 tablet_replica.cc:331] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: stopping tablet replica
I20250110 03:07:58.598135 5850 raft_consensus.cc:2238] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:58.598214 5850 raft_consensus.cc:2267] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:58.598495 5850 ts_tablet_manager.cc:1905] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:58.599697 5850 ts_tablet_manager.cc:1918] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:58.599777 5850 log.cc:1198] T 56330dd8a2ef4597be2d809c1cf02000 P b680ae46493d4601a57d0545c65a1895: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/56330dd8a2ef4597be2d809c1cf02000
I20250110 03:07:58.600178 4225 catalog_manager.cc:4872] TS b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061): tablet 56330dd8a2ef4597be2d809c1cf02000 (table rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]) successfully deleted
I20250110 03:07:58.647238 5569 raft_consensus.cc:1059] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:07:58.647375 5569 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:58.647822 4381 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.647876 4513 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.647877 5235 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.648053 5569 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.648168 5804 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.648293 5771 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.649044 5804 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.649205 4381 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.649245 5235 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.650287 4236 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 reported cstate change: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.650323 4513 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.651319 4646 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:58.651700 4381 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.651718 5235 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.652009 5836 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.652098 5771 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.652667 5804 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.652784 4381 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.653420 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8953c1791507457e805dd8a89638dd29 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:58.653620 4235 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:58.653872 5235 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.654368 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet 8953c1791507457e805dd8a89638dd29 with delete_type TABLET_DATA_TOMBSTONED (TS e8ba4f91a2c94c3883d60f7002e12bb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51178
I20250110 03:07:58.654487 5844 tablet_replica.cc:331] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: stopping tablet replica
I20250110 03:07:58.654587 5844 raft_consensus.cc:2238] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:58.654649 5844 raft_consensus.cc:2267] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:58.654907 5844 ts_tablet_manager.cc:1905] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:58.655928 5844 ts_tablet_manager.cc:1918] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:58.655987 5844 log.cc:1198] T 8953c1791507457e805dd8a89638dd29 P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/8953c1791507457e805dd8a89638dd29
I20250110 03:07:58.656291 4224 catalog_manager.cc:4872] TS e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399): tablet 8953c1791507457e805dd8a89638dd29 (table rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]) successfully deleted
I20250110 03:07:58.662716 5846 raft_consensus.cc:1059] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:07:58.662855 5846 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:58.663280 4646 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.663278 5366 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.663290 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.663477 5579 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.663575 5770 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.663689 5772 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.664429 5579 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.664579 5366 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.664772 4381 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.665338 4646 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.665747 4233 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:58.666947 4513 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:58.667291 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.667369 5366 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.667467 5772 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.667555 5846 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.668092 5770 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.668196 4381 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.668956 4224 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:58.669004 4240 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:58.669114 5366 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.669636 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with delete_type TABLET_DATA_TOMBSTONED (TS b680ae46493d4601a57d0545c65a1895 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57156
I20250110 03:07:58.669744 5850 tablet_replica.cc:331] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: stopping tablet replica
I20250110 03:07:58.669802 5850 raft_consensus.cc:2238] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:58.669854 5850 raft_consensus.cc:2267] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:58.670123 5850 ts_tablet_manager.cc:1905] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:58.671144 5850 ts_tablet_manager.cc:1918] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:58.671190 5850 log.cc:1198] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P b680ae46493d4601a57d0545c65a1895: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/e5cecae3ae274ba9bcdf8df3a8ebdf46
I20250110 03:07:58.671478 4225 catalog_manager.cc:4872] TS b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061): tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 (table rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]) successfully deleted
I20250110 03:07:58.690313 5771 raft_consensus.cc:1059] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:07:58.690457 5771 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:58.690934 4513 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.690954 5235 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.690953 4381 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.691165 5569 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.691251 5771 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.691306 5804 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.692107 5836 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.692229 4513 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.692253 4381 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.693075 4236 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.693171 5235 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.693830 4646 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:58.694326 4381 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.694353 5235 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.694518 5804 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.694620 5836 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.695232 5569 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.695876 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ec019c792274403585d757840bdf84aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:58.696008 5235 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.696004 4235 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:58.696203 4381 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:58.697142 4492 tablet_service.cc:1514] Processing DeleteTablet for tablet ec019c792274403585d757840bdf84aa with delete_type TABLET_DATA_TOMBSTONED (TS e8ba4f91a2c94c3883d60f7002e12bb2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51178
I20250110 03:07:58.697252 5844 tablet_replica.cc:331] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: stopping tablet replica
I20250110 03:07:58.697322 5844 raft_consensus.cc:2238] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:58.697376 5844 raft_consensus.cc:2267] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:58.697681 5844 ts_tablet_manager.cc:1905] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:58.698772 5844 ts_tablet_manager.cc:1918] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:58.698827 5844 log.cc:1198] T ec019c792274403585d757840bdf84aa P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-1/wal/wals/ec019c792274403585d757840bdf84aa
I20250110 03:07:58.699113 4224 catalog_manager.cc:4872] TS e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399): tablet ec019c792274403585d757840bdf84aa (table rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]) successfully deleted
I20250110 03:07:58.804899 5913 raft_consensus.cc:1059] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:07:58.805100 5913 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:58.805634 5366 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.805634 4646 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.805675 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:07:58.805856 5916 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.805966 5913 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.806073 5915 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:07:58.807117 5915 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.807257 5366 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.807965 4646 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.808137 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.809026 4236 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 4 to 5, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.809931 4381 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:58.810288 5366 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.810297 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:07:58.810469 5913 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.810566 5916 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:07:58.811141 5916 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.811239 5366 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.811944 4224 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:07:58.812144 4235 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.812253 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b680ae46493d4601a57d0545c65a1895 (127.14.19.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:58.812847 4625 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with delete_type TABLET_DATA_TOMBSTONED (TS b680ae46493d4601a57d0545c65a1895 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57156
I20250110 03:07:58.812948 5850 tablet_replica.cc:331] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: stopping tablet replica
I20250110 03:07:58.813020 5850 raft_consensus.cc:2238] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:07:58.813072 5850 raft_consensus.cc:2267] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:07:58.813338 5850 ts_tablet_manager.cc:1905] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:07:58.814271 5850 ts_tablet_manager.cc:1918] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250110 03:07:58.814318 5850 log.cc:1198] T 1cd35b6ae5b54b05bb6d93b79442ba9d P b680ae46493d4601a57d0545c65a1895: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-2/wal/wals/1cd35b6ae5b54b05bb6d93b79442ba9d
I20250110 03:07:58.814637 4225 catalog_manager.cc:4872] TS b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061): tablet 1cd35b6ae5b54b05bb6d93b79442ba9d (table rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]) successfully deleted
I20250110 03:07:58.870082 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:58.870548 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:58.872113 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:58.876557 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:58.878777 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:58.911295 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:58.912447 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:58.912940 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:58.913280 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:58.920102 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:58.948895 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:58.948901 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:58.950361 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:58.950631 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:58.957769 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:59.054900 4240 ts_manager.cc:292] Set tserver state for e3a520ce69d74074a3d309fd843bccd6 to MAINTENANCE_MODE
I20250110 03:07:59.163597 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:59.164057 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:59.166044 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:59.167905 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:59.171334 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:59.207825 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:59.208518 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:59.211067 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:59.212028 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:59.216972 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:59.266345 5346 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:59.266722 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:07:59.267549 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:07:59.273468 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:07:59.276580 4361 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:07:59.285446 4646 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:59.285843 5235 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:59.285848 4381 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:59.286064 6154 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:59.286207 5836 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:59.286927 6154 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.287057 5235 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.287825 4381 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.288023 4240 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:59.289633 4646 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:59.290105 4381 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:59.290185 5235 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:59.290272 5836 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:59.290637 5836 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:59.290822 6155 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:59.291201 4381 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:59.291801 4225 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8953c1791507457e805dd8a89638dd29 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:07:59.291826 4233 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:59.292284 5235 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:59.293715 4381 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:07:59.294099 5366 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:59.294121 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:59.294296 5803 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 03:07:59.294406 4580 consensus_peers.cc:487] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8953c1791507457e805dd8a89638dd29. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:59.294442 5803 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:59.294934 5803 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:59.295068 5366 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:59.295802 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:07:59.295897 4240 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:07:59.296746 4381 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:59.297173 5366 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:59.297432 5915 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:59.297711 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:59.297885 5915 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:59.298233 5915 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:59.298597 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:59.299082 4224 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:07:59.299188 5366 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:07:59.299607 4233 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
W20250110 03:07:59.300714 4315 consensus_peers.cc:487] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cd35b6ae5b54b05bb6d93b79442ba9d. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:59.302723 4646 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:07:59.303084 5235 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:59.303128 4381 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:07:59.303308 6154 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:59.303398 6155 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:07:59.303943 5836 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.304040 5235 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.304064 4381 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.304973 4240 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:07:59.306196 4646 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:59.306597 5235 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:07:59.306659 4580 consensus_peers.cc:487] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec019c792274403585d757840bdf84aa. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:07:59.306777 6155 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:59.307053 4381 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:07:59.307209 6155 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:07:59.307322 5836 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:59.308049 4225 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ec019c792274403585d757840bdf84aa with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:07:59.308118 4381 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:59.308351 4233 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:07:59.308557 5235 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:07:59.768491 6214 ts_tablet_manager.cc:927] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651)
I20250110 03:07:59.768666 6214 tablet_copy_client.cc:323] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.129:37651
I20250110 03:07:59.771350 4402 tablet_copy_service.cc:140] P e3a520ce69d74074a3d309fd843bccd6: Received BeginTabletCopySession request for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:35319)
I20250110 03:07:59.771412 4402 tablet_copy_service.cc:161] P e3a520ce69d74074a3d309fd843bccd6: Beginning new tablet copy session on tablet 1cd35b6ae5b54b05bb6d93b79442ba9d from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:35319: session id = 8cb2e4afb3384e40879d605ea471b039-1cd35b6ae5b54b05bb6d93b79442ba9d
I20250110 03:07:59.771908 4402 tablet_copy_source_session.cc:215] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:59.772282 6214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:59.773336 6214 tablet_copy_client.cc:806] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:59.773455 6214 tablet_copy_client.cc:670] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:59.774128 6214 tablet_copy_client.cc:538] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:59.774719 6214 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:07:59.778340 6214 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:59.778510 6214 tablet_bootstrap.cc:492] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:07:59.778614 6214 ts_tablet_manager.cc:1397] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250110 03:07:59.778733 6214 raft_consensus.cc:357] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:59.778791 6214 raft_consensus.cc:738] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:07:59.778824 6214 consensus_queue.cc:260] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:07:59.778918 6214 ts_tablet_manager.cc:1428] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:59.779143 4402 tablet_copy_service.cc:342] P e3a520ce69d74074a3d309fd843bccd6: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-1cd35b6ae5b54b05bb6d93b79442ba9d received from {username='slave'} at 127.14.19.133:35319
I20250110 03:07:59.779191 4402 tablet_copy_service.cc:434] P e3a520ce69d74074a3d309fd843bccd6: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-1cd35b6ae5b54b05bb6d93b79442ba9d on tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:07:59.885254 6219 ts_tablet_manager.cc:927] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:59.885430 6219 tablet_copy_client.cc:323] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:07:59.888134 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet 8953c1791507457e805dd8a89638dd29 from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:44479)
I20250110 03:07:59.888196 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet 8953c1791507457e805dd8a89638dd29 from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:44479: session id = 22901baa49564439bfd6978fe2e4a4a9-8953c1791507457e805dd8a89638dd29
I20250110 03:07:59.888573 4666 tablet_copy_source_session.cc:215] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:59.888901 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8953c1791507457e805dd8a89638dd29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:59.889930 6219 tablet_copy_client.cc:806] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:59.890056 6219 tablet_copy_client.cc:670] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:59.890678 6219 tablet_copy_client.cc:538] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:59.891209 6219 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:07:59.894344 6219 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:59.894528 6219 tablet_bootstrap.cc:492] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:07:59.894639 6219 ts_tablet_manager.cc:1397] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.003s user 0.004s sys 0.000s
I20250110 03:07:59.894780 6219 raft_consensus.cc:357] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:59.894865 6219 raft_consensus.cc:738] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:07:59.894889 6219 consensus_queue.cc:260] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:59.895069 6219 ts_tablet_manager.cc:1428] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:59.895277 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-8953c1791507457e805dd8a89638dd29 received from {username='slave'} at 127.14.19.132:44479
I20250110 03:07:59.895336 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-8953c1791507457e805dd8a89638dd29 on tablet 8953c1791507457e805dd8a89638dd29 with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:07:59.918150 6219 ts_tablet_manager.cc:927] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:07:59.918283 6219 tablet_copy_client.cc:323] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:07:59.918484 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet ec019c792274403585d757840bdf84aa from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:44479)
I20250110 03:07:59.918534 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet ec019c792274403585d757840bdf84aa from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:44479: session id = 22901baa49564439bfd6978fe2e4a4a9-ec019c792274403585d757840bdf84aa
I20250110 03:07:59.918879 4666 tablet_copy_source_session.cc:215] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:07:59.919162 6219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec019c792274403585d757840bdf84aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:07:59.919864 6219 tablet_copy_client.cc:806] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:07:59.919974 6219 tablet_copy_client.cc:670] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:07:59.920531 6219 tablet_copy_client.cc:538] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:07:59.921029 6219 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:07:59.923794 6219 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:07:59.923962 6219 tablet_bootstrap.cc:492] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:07:59.924058 6219 ts_tablet_manager.cc:1397] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 03:07:59.924139 6219 raft_consensus.cc:357] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:59.924201 6219 raft_consensus.cc:738] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:07:59.924223 6219 consensus_queue.cc:260] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:07:59.924311 6219 ts_tablet_manager.cc:1428] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:07:59.924505 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-ec019c792274403585d757840bdf84aa received from {username='slave'} at 127.14.19.132:44479
I20250110 03:07:59.924563 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-ec019c792274403585d757840bdf84aa on tablet ec019c792274403585d757840bdf84aa with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:08:00.307547 5235 raft_consensus.cc:1212] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:00.369575 5366 raft_consensus.cc:1212] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:00.495126 5366 raft_consensus.cc:1212] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:00.818177 5803 raft_consensus.cc:1059] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:08:00.818346 5803 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:00.818841 5235 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:00.818862 5366 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:00.818856 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer e3a520ce69d74074a3d309fd843bccd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:00.819094 6305 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:00.819265 6308 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:00.819393 6307 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:00.820219 6308 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:00.820338 5366 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:00.820389 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:00.821467 4233 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3a520ce69d74074a3d309fd843bccd6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:00.821530 5235 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:00.885643 6218 raft_consensus.cc:1059] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:08:00.885841 6218 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:00.886335 4381 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:00.886348 5366 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:00.886356 5235 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:00.886575 6318 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:00.886720 6273 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:00.886839 6218 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:00.887435 6273 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.887607 5235 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.887686 5366 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.888792 4235 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:00.888842 4381 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.889768 4646 consensus_queue.cc:237] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:00.890132 5366 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:00.890233 5235 raft_consensus.cc:1270] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:00.890307 6218 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:00.890532 6218 consensus_queue.cc:1035] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:00.891039 6318 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.891157 5366 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.891252 5235 raft_consensus.cc:2949] T 8953c1791507457e805dd8a89638dd29 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.892020 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8953c1791507457e805dd8a89638dd29 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:00.892100 4236 catalog_manager.cc:5526] T 8953c1791507457e805dd8a89638dd29 P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:00.892899 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 8953c1791507457e805dd8a89638dd29 with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:00.893124 6322 tablet_replica.cc:331] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:00.893232 6322 raft_consensus.cc:2238] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:00.893332 6322 raft_consensus.cc:2267] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:00.893635 6322 ts_tablet_manager.cc:1905] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:00.894893 6322 ts_tablet_manager.cc:1918] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:08:00.894956 6322 log.cc:1198] T 8953c1791507457e805dd8a89638dd29 P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/8953c1791507457e805dd8a89638dd29
I20250110 03:08:00.895304 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet 8953c1791507457e805dd8a89638dd29 (table rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]) successfully deleted
I20250110 03:08:01.007051 4381 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d"
dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45032
I20250110 03:08:01.007133 4381 raft_consensus.cc:604] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:08:01.061399 6273 raft_consensus.cc:1059] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:08:01.061553 6273 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:01.062047 5366 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:01.062084 5235 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:01.062098 4381 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:01.062297 6218 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:01.062407 6318 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:01.062557 6274 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:01.063453 6274 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.063617 4381 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.063751 5235 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.064640 4235 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:01.064656 5366 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.065567 4646 consensus_queue.cc:237] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:01.065987 5366 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:01.066143 6318 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:01.066212 5235 raft_consensus.cc:1270] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:01.066362 6318 consensus_queue.cc:1035] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:01.066699 6218 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.066815 5366 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.066988 5235 raft_consensus.cc:2949] T ec019c792274403585d757840bdf84aa P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.067561 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ec019c792274403585d757840bdf84aa with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:01.067796 4236 catalog_manager.cc:5526] T ec019c792274403585d757840bdf84aa P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.068475 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet ec019c792274403585d757840bdf84aa with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:01.068574 6322 tablet_replica.cc:331] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:01.068634 6322 raft_consensus.cc:2238] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:01.068724 6322 raft_consensus.cc:2267] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:01.069013 6322 ts_tablet_manager.cc:1905] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:01.070168 6322 ts_tablet_manager.cc:1918] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:08:01.070226 6322 log.cc:1198] T ec019c792274403585d757840bdf84aa P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/ec019c792274403585d757840bdf84aa
I20250110 03:08:01.070600 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet ec019c792274403585d757840bdf84aa (table rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]) successfully deleted
I20250110 03:08:01.211776 4513 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:01.212272 5366 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:01.212411 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:01.212464 6354 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:01.212605 5846 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:01.213174 5846 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.213303 5366 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.213383 4381 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:01.214571 4233 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:01.215662 4513 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:01.216095 5366 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:08:01.216312 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:08:01.216490 6354 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:01.216578 5846 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:01.217185 6362 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:01.217283 4381 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:01.218325 4233 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:08:01.218587 4224 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:08:01.218606 5366 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
W20250110 03:08:01.220810 4447 consensus_peers.cc:487] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5cecae3ae274ba9bcdf8df3a8ebdf46. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:01.228571 4381 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d"
dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45054
I20250110 03:08:01.228631 4381 raft_consensus.cc:604] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Received request to transfer leadership
I20250110 03:08:01.283128 6305 raft_consensus.cc:988] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: : Instructing follower 22901baa49564439bfd6978fe2e4a4a9 to start an election
I20250110 03:08:01.283213 6305 raft_consensus.cc:1076] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Signalling peer 22901baa49564439bfd6978fe2e4a4a9 to start an election
I20250110 03:08:01.283433 5366 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d"
dest_uuid: "22901baa49564439bfd6978fe2e4a4a9"
from {username='slave'} at 127.14.19.129:47211
I20250110 03:08:01.283536 5366 raft_consensus.cc:491] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 03:08:01.283591 5366 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:08:01.284173 5366 raft_consensus.cc:513] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:01.284415 5366 leader_election.cc:290] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [CANDIDATE]: Term 2 election: Requested vote from peers e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651), e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399), 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025)
I20250110 03:08:01.284572 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "22901baa49564439bfd6978fe2e4a4a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e3a520ce69d74074a3d309fd843bccd6"
I20250110 03:08:01.284595 4513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "22901baa49564439bfd6978fe2e4a4a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2"
I20250110 03:08:01.284649 4381 raft_consensus.cc:3049] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 03:08:01.284658 4513 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:08:01.284672 4381 raft_consensus.cc:738] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3a520ce69d74074a3d309fd843bccd6, State: Running, Role: LEADER
I20250110 03:08:01.284731 4381 consensus_queue.cc:260] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:01.284839 4381 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:08:01.285396 4513 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22901baa49564439bfd6978fe2e4a4a9 in term 2.
I20250110 03:08:01.285481 4381 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22901baa49564439bfd6978fe2e4a4a9 in term 2.
I20250110 03:08:01.285626 5299 leader_election.cc:304] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 22901baa49564439bfd6978fe2e4a4a9, e3a520ce69d74074a3d309fd843bccd6, e8ba4f91a2c94c3883d60f7002e12bb2; no voters:
I20250110 03:08:01.285775 6223 raft_consensus.cc:2798] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 03:08:01.285912 6223 raft_consensus.cc:695] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 2 LEADER]: Becoming Leader. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Running, Role: LEADER
I20250110 03:08:01.286005 6223 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:01.286665 4235 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: term changed from 1 to 2, leader changed from e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) to 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132). New cstate: current_term: 2 leader_uuid: "22901baa49564439bfd6978fe2e4a4a9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:08:01.287467 5235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd35b6ae5b54b05bb6d93b79442ba9d" candidate_uuid: "22901baa49564439bfd6978fe2e4a4a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8cb2e4afb3384e40879d605ea471b039"
I20250110 03:08:01.287550 5235 raft_consensus.cc:3054] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Advancing to term 2
I20250110 03:08:01.288101 5235 raft_consensus.cc:2463] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22901baa49564439bfd6978fe2e4a4a9 in term 2.
I20250110 03:08:01.433596 4646 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:08:01.434167 4381 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:08:01.434176 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:08:01.434407 6273 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:08:01.434511 6274 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:08:01.435295 6318 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:01.435534 4381 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:01.435585 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:01.436676 4235 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:01.437705 4646 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:01.438149 4381 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:08:01.438199 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:08:01.438397 4579 consensus_peers.cc:487] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b6206c7b3014154ba2689f574532355. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:01.438489 6274 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:08:01.438616 6218 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:08:01.439175 6274 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:01.439422 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:01.439898 4381 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:01.440374 4225 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8b6206c7b3014154ba2689f574532355 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:08:01.440652 4233 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:01.679517 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 2 FOLLOWER]: Refusing update from remote peer 22901baa49564439bfd6978fe2e4a4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:08:01.679778 6223 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:01.691036 6421 ts_tablet_manager.cc:927] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:08:01.691203 6421 tablet_copy_client.cc:323] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:43399
I20250110 03:08:01.693625 4534 tablet_copy_service.cc:140] P e8ba4f91a2c94c3883d60f7002e12bb2: Received BeginTabletCopySession request for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39839)
I20250110 03:08:01.693676 4534 tablet_copy_service.cc:161] P e8ba4f91a2c94c3883d60f7002e12bb2: Beginning new tablet copy session on tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39839: session id = 8cb2e4afb3384e40879d605ea471b039-e5cecae3ae274ba9bcdf8df3a8ebdf46
I20250110 03:08:01.694059 4534 tablet_copy_source_session.cc:215] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:01.694370 6421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:01.695248 6421 tablet_copy_client.cc:806] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:01.695354 6421 tablet_copy_client.cc:670] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:01.695899 6421 tablet_copy_client.cc:538] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:01.696382 6421 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:08:01.699126 6421 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:01.699293 6421 tablet_bootstrap.cc:492] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:08:01.699381 6421 ts_tablet_manager.cc:1397] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:08:01.699478 6421 raft_consensus.cc:357] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:01.699535 6421 raft_consensus.cc:738] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:08:01.699568 6421 consensus_queue.cc:260] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:01.699729 6421 ts_tablet_manager.cc:1428] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20250110 03:08:01.699919 4534 tablet_copy_service.cc:342] P e8ba4f91a2c94c3883d60f7002e12bb2: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-e5cecae3ae274ba9bcdf8df3a8ebdf46 received from {username='slave'} at 127.14.19.133:39839
I20250110 03:08:01.699978 4534 tablet_copy_service.cc:434] P e8ba4f91a2c94c3883d60f7002e12bb2: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-e5cecae3ae274ba9bcdf8df3a8ebdf46 on tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:08:01.793964 5235 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 2 FOLLOWER]: Refusing update from remote peer 22901baa49564439bfd6978fe2e4a4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:08:01.794224 6223 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:01.795861 4381 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 2 FOLLOWER]: Refusing update from remote peer 22901baa49564439bfd6978fe2e4a4a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 03:08:01.796211 6426 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:01.798903 5366 consensus_queue.cc:237] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:01.799300 5235 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 2 FOLLOWER]: Refusing update from remote peer 22901baa49564439bfd6978fe2e4a4a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:08:01.799319 4513 raft_consensus.cc:1270] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 2 FOLLOWER]: Refusing update from remote peer 22901baa49564439bfd6978fe2e4a4a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250110 03:08:01.799480 6426 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:08:01.799546 6429 consensus_queue.cc:1035] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250110 03:08:01.800156 6223 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 22901baa49564439bfd6978fe2e4a4a9 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:01.800293 4513 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:01.800307 5235 raft_consensus.cc:2949] T 1cd35b6ae5b54b05bb6d93b79442ba9d P 8cb2e4afb3384e40879d605ea471b039 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:01.801316 4235 catalog_manager.cc:5526] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 9 to 11, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 2 leader_uuid: "22901baa49564439bfd6978fe2e4a4a9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:01.801383 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:01.802197 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd35b6ae5b54b05bb6d93b79442ba9d with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:01.802390 6433 tablet_replica.cc:331] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:01.802464 6433 raft_consensus.cc:2238] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:01.802510 6433 raft_consensus.cc:2267] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:01.802793 6433 ts_tablet_manager.cc:1905] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:01.804143 6433 ts_tablet_manager.cc:1918] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20250110 03:08:01.804193 6433 log.cc:1198] T 1cd35b6ae5b54b05bb6d93b79442ba9d P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/1cd35b6ae5b54b05bb6d93b79442ba9d
I20250110 03:08:01.804526 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet 1cd35b6ae5b54b05bb6d93b79442ba9d (table rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]) successfully deleted
I20250110 03:08:02.028491 6421 ts_tablet_manager.cc:927] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:08:02.028694 6421 tablet_copy_client.cc:323] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:08:02.028973 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet 8b6206c7b3014154ba2689f574532355 from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39641)
I20250110 03:08:02.029031 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet 8b6206c7b3014154ba2689f574532355 from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39641: session id = 8cb2e4afb3384e40879d605ea471b039-8b6206c7b3014154ba2689f574532355
I20250110 03:08:02.029433 4666 tablet_copy_source_session.cc:215] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:02.029773 6421 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6206c7b3014154ba2689f574532355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:02.030846 6421 tablet_copy_client.cc:806] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:02.030953 6421 tablet_copy_client.cc:670] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:02.031461 6421 tablet_copy_client.cc:538] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:02.032001 6421 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:08:02.034684 6421 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:02.034848 6421 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:08:02.034933 6421 ts_tablet_manager.cc:1397] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20250110 03:08:02.035048 6421 raft_consensus.cc:357] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:02.035104 6421 raft_consensus.cc:738] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:08:02.035136 6421 consensus_queue.cc:260] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:02.035225 6421 ts_tablet_manager.cc:1428] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:08:02.035421 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-8b6206c7b3014154ba2689f574532355 received from {username='slave'} at 127.14.19.133:39641
I20250110 03:08:02.035475 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-8b6206c7b3014154ba2689f574532355 on tablet 8b6206c7b3014154ba2689f574532355 with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:08:02.075562 4646 consensus_queue.cc:237] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } }
I20250110 03:08:02.075999 4381 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:08:02.076022 4513 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:08:02.076234 6218 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:08:02.076388 6457 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:08:02.077171 6218 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:02.077344 4381 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:02.078266 4513 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } }
I20250110 03:08:02.078428 4233 catalog_manager.cc:5526] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } }
I20250110 03:08:02.079411 4646 consensus_queue.cc:237] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:08:02.079972 4381 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:08:02.079982 4513 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20250110 03:08:02.080150 4580 consensus_peers.cc:487] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef7d7b7f5c444da0bfba41db5a231c30. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:02.080165 6218 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:08:02.080399 6457 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:08:02.081017 6218 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:02.081176 4381 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:02.082165 4225 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ef7d7b7f5c444da0bfba41db5a231c30 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:08:02.082216 4240 catalog_manager.cc:5526] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 4 to 5, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:08:02.082512 4513 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:02.256713 5235 raft_consensus.cc:1212] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:02.422792 5235 raft_consensus.cc:1212] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:08:02.496508 6494 ts_tablet_manager.cc:927] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:08:02.496652 6494 tablet_copy_client.cc:323] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:08:02.496886 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet ef7d7b7f5c444da0bfba41db5a231c30 from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:44479)
I20250110 03:08:02.496949 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet ef7d7b7f5c444da0bfba41db5a231c30 from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:44479: session id = 22901baa49564439bfd6978fe2e4a4a9-ef7d7b7f5c444da0bfba41db5a231c30
I20250110 03:08:02.497335 4666 tablet_copy_source_session.cc:215] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:02.497677 6494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef7d7b7f5c444da0bfba41db5a231c30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:02.498607 6494 tablet_copy_client.cc:806] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:02.498715 6494 tablet_copy_client.cc:670] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:02.499282 6494 tablet_copy_client.cc:538] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:02.499836 6494 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:08:02.503027 6494 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:02.503206 6494 tablet_bootstrap.cc:492] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:08:02.503307 6494 ts_tablet_manager.cc:1397] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:08:02.503420 6494 raft_consensus.cc:357] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:08:02.503492 6494 raft_consensus.cc:738] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:08:02.503522 6494 consensus_queue.cc:260] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:08:02.503612 6494 ts_tablet_manager.cc:1428] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:08:02.503814 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-ef7d7b7f5c444da0bfba41db5a231c30 received from {username='slave'} at 127.14.19.132:44479
I20250110 03:08:02.503868 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-ef7d7b7f5c444da0bfba41db5a231c30 on tablet ef7d7b7f5c444da0bfba41db5a231c30 with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:08:02.735467 6478 raft_consensus.cc:1059] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:08:02.735648 6478 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:02.736137 5235 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:02.736171 5365 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:02.736188 4381 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:02.736330 6528 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:02.736441 6529 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:02.736560 6354 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:02.737506 6478 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.738531 5235 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.738610 4381 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.738659 4240 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:02.738466 5365 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.740777 4513 consensus_queue.cc:237] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:02.741145 5365 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:02.741153 5235 raft_consensus.cc:1270] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:02.741308 6529 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:02.741401 6478 consensus_queue.cc:1035] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:02.741923 6354 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.742110 5365 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.742115 5235 raft_consensus.cc:2949] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:02.742982 4224 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:02.743064 4233 catalog_manager.cc:5526] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:02.744990 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:02.745153 6537 tablet_replica.cc:331] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:02.745203 6537 raft_consensus.cc:2238] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:02.745266 6537 raft_consensus.cc:2267] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:02.745479 6537 ts_tablet_manager.cc:1905] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:02.746632 6537 ts_tablet_manager.cc:1918] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:08:02.746678 6537 log.cc:1198] T e5cecae3ae274ba9bcdf8df3a8ebdf46 P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/e5cecae3ae274ba9bcdf8df3a8ebdf46
I20250110 03:08:02.746982 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet e5cecae3ae274ba9bcdf8df3a8ebdf46 (table rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]) successfully deleted
I20250110 03:08:03.009384 6460 raft_consensus.cc:1059] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:08:03.009544 6460 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:03.009989 5235 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:03.010042 4381 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:03.010030 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:03.010233 6460 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:03.010346 6553 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:03.010509 6555 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:03.011374 5235 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.011478 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.011271 6553 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.012672 4235 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 reported cstate change: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.013653 4381 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.013896 4646 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:03.014242 5235 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.014333 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.014423 6460 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.014680 6460 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.015065 6559 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.015223 5235 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.015446 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.016366 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8b6206c7b3014154ba2689f574532355 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:03.016515 4233 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.017220 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6206c7b3014154ba2689f574532355 with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:03.017319 6537 tablet_replica.cc:331] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:03.017374 6537 raft_consensus.cc:2238] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:03.017437 6537 raft_consensus.cc:2267] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:03.017715 6537 ts_tablet_manager.cc:1905] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:03.018739 6537 ts_tablet_manager.cc:1918] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:08:03.018784 6537 log.cc:1198] T 8b6206c7b3014154ba2689f574532355 P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b6206c7b3014154ba2689f574532355
I20250110 03:08:03.019106 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet 8b6206c7b3014154ba2689f574532355 (table rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]) successfully deleted
I20250110 03:08:03.113370 5365 raft_consensus.cc:1212] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:08:03.136286 4646 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:03.136765 5235 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.136801 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.136976 6460 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.137094 6559 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.137811 6559 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.137986 5235 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.138075 4381 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.139112 4235 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:03.140121 4646 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } }
I20250110 03:08:03.140514 5235 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:08:03.140514 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:08:03.140738 6555 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:03.140815 6460 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250110 03:08:03.140920 4579 consensus_peers.cc:487] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 -> Peer e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399): Couldn't send request to peer e8ba4f91a2c94c3883d60f7002e12bb2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:03.141431 6460 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } } }
I20250110 03:08:03.141697 5235 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } } }
I20250110 03:08:03.142258 4381 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } } }
I20250110 03:08:03.142453 4225 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a99a80262a3d492d838cc0e0fc757ffc with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:08:03.142643 4233 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 reported cstate change: config changed from index 7 to 8, NON_VOTER e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } } }
I20250110 03:08:03.354643 4513 consensus_queue.cc:237] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } }
I20250110 03:08:03.355161 4381 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:08:03.355208 4646 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250110 03:08:03.355422 6589 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:08:03.355562 6478 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250110 03:08:03.356184 6589 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } }
I20250110 03:08:03.356341 4381 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } }
I20250110 03:08:03.356526 4646 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } }
I20250110 03:08:03.357509 4235 catalog_manager.cc:5526] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } }
I20250110 03:08:03.358409 4513 consensus_queue.cc:237] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:03.358831 4646 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:08:03.358897 4381 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250110 03:08:03.359082 6592 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 03:08:03.359155 6589 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250110 03:08:03.359278 4447 consensus_peers.cc:487] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f78c4d4f4de4b83b2b63a3b31e20fae. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:03.359676 6592 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:03.360631 4224 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:08:03.360699 4381 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:03.360786 4646 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:03.360818 4233 catalog_manager.cc:5526] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 4 to 5, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 03:08:03.547058 6608 ts_tablet_manager.cc:927] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:08:03.547286 6608 tablet_copy_client.cc:323] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:08:03.547610 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet a99a80262a3d492d838cc0e0fc757ffc from peer e8ba4f91a2c94c3883d60f7002e12bb2 ({username='slave'} at 127.14.19.130:36659)
I20250110 03:08:03.547693 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet a99a80262a3d492d838cc0e0fc757ffc from peer e8ba4f91a2c94c3883d60f7002e12bb2 at {username='slave'} at 127.14.19.130:36659: session id = e8ba4f91a2c94c3883d60f7002e12bb2-a99a80262a3d492d838cc0e0fc757ffc
I20250110 03:08:03.548079 4666 tablet_copy_source_session.cc:215] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:03.548414 6608 ts_tablet_manager.cc:1905] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 03:08:03.549878 6608 ts_tablet_manager.cc:1918] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20250110 03:08:03.549979 6608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a99a80262a3d492d838cc0e0fc757ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:03.550501 6608 tablet_copy_client.cc:806] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:03.550644 6608 tablet_copy_client.cc:670] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:03.551311 6608 tablet_copy_client.cc:538] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:03.551843 6608 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap starting.
I20250110 03:08:03.554800 6608 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:03.554992 6608 tablet_bootstrap.cc:492] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Bootstrap complete.
I20250110 03:08:03.555083 6608 ts_tablet_manager.cc:1397] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:08:03.555189 6608 raft_consensus.cc:357] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } }
I20250110 03:08:03.555248 6608 raft_consensus.cc:738] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e8ba4f91a2c94c3883d60f7002e12bb2, State: Initialized, Role: LEARNER
I20250110 03:08:03.555279 6608 consensus_queue.cc:260] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: NON_VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: true } }
I20250110 03:08:03.555378 6608 ts_tablet_manager.cc:1428] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:08:03.555581 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session e8ba4f91a2c94c3883d60f7002e12bb2-a99a80262a3d492d838cc0e0fc757ffc received from {username='slave'} at 127.14.19.130:36659
I20250110 03:08:03.555629 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session e8ba4f91a2c94c3883d60f7002e12bb2-a99a80262a3d492d838cc0e0fc757ffc on tablet a99a80262a3d492d838cc0e0fc757ffc with peer e8ba4f91a2c94c3883d60f7002e12bb2
I20250110 03:08:03.650810 6553 raft_consensus.cc:1059] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:08:03.650987 6553 consensus_queue.cc:237] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:03.651388 5365 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:03.651482 4381 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:03.651525 4513 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:03.651628 6628 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:03.651752 6626 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:03.651827 6629 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:03.652618 6553 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.652769 5365 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.652760 4381 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.653990 4233 catalog_manager.cc:5526] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 5 to 6, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.653990 4513 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.655239 4646 consensus_queue.cc:237] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:03.655615 5365 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.655740 4513 raft_consensus.cc:1270] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.655808 6626 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.655920 6629 consensus_queue.cc:1035] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.656424 6626 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.656577 4513 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.657277 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ef7d7b7f5c444da0bfba41db5a231c30 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:03.657361 4240 catalog_manager.cc:5526] T ef7d7b7f5c444da0bfba41db5a231c30 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:03.657589 5365 raft_consensus.cc:2949] T ef7d7b7f5c444da0bfba41db5a231c30 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.658564 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7d7b7f5c444da0bfba41db5a231c30 with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:03.658807 6633 tablet_replica.cc:331] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:03.658890 6633 raft_consensus.cc:2238] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:03.658962 6633 raft_consensus.cc:2267] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:03.659180 6633 ts_tablet_manager.cc:1905] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:03.660245 6633 ts_tablet_manager.cc:1918] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:08:03.660291 6633 log.cc:1198] T ef7d7b7f5c444da0bfba41db5a231c30 P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef7d7b7f5c444da0bfba41db5a231c30
I20250110 03:08:03.660579 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet ef7d7b7f5c444da0bfba41db5a231c30 (table rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]) successfully deleted
I20250110 03:08:03.958281 6649 ts_tablet_manager.cc:927] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:08:03.958417 6649 tablet_copy_client.cc:323] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:43399
I20250110 03:08:03.958647 4534 tablet_copy_service.cc:140] P e8ba4f91a2c94c3883d60f7002e12bb2: Received BeginTabletCopySession request for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39839)
I20250110 03:08:03.958710 4534 tablet_copy_service.cc:161] P e8ba4f91a2c94c3883d60f7002e12bb2: Beginning new tablet copy session on tablet 5f78c4d4f4de4b83b2b63a3b31e20fae from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39839: session id = 8cb2e4afb3384e40879d605ea471b039-5f78c4d4f4de4b83b2b63a3b31e20fae
I20250110 03:08:03.959095 4534 tablet_copy_source_session.cc:215] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:03.959484 6649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:03.960351 6649 tablet_copy_client.cc:806] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:03.960482 6649 tablet_copy_client.cc:670] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:03.961061 6649 tablet_copy_client.cc:538] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:03.961551 6649 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:08:03.964249 6649 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:03.964406 6649 tablet_bootstrap.cc:492] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:08:03.964475 6649 ts_tablet_manager.cc:1397] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250110 03:08:03.964567 6649 raft_consensus.cc:357] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:03.964623 6649 raft_consensus.cc:738] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:08:03.964644 6649 consensus_queue.cc:260] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:03.964754 6649 ts_tablet_manager.cc:1428] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20250110 03:08:03.964941 4534 tablet_copy_service.cc:342] P e8ba4f91a2c94c3883d60f7002e12bb2: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-5f78c4d4f4de4b83b2b63a3b31e20fae received from {username='slave'} at 127.14.19.133:39839
I20250110 03:08:03.964999 4534 tablet_copy_service.cc:434] P e8ba4f91a2c94c3883d60f7002e12bb2: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-5f78c4d4f4de4b83b2b63a3b31e20fae on tablet 5f78c4d4f4de4b83b2b63a3b31e20fae with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:08:03.993423 4513 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:03.993891 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.993889 5365 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:03.994160 6659 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.994279 6592 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:03.994899 6659 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.995086 5365 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.995800 4381 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:03.996176 4233 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:03.997159 4513 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:03.997671 5365 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:08:03.997773 4447 consensus_peers.cc:487] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 -> Peer 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): Couldn't send request to peer 8cb2e4afb3384e40879d605ea471b039. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56330dd8a2ef4597be2d809c1cf02000. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:03.997874 6668 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:03.997983 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:08:03.998241 6668 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:03.998391 6668 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:03.998500 5365 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:03.999380 4224 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 56330dd8a2ef4597be2d809c1cf02000 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250110 03:08:03.999431 4240 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:04.000166 4381 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } } }
I20250110 03:08:04.045042 4513 raft_consensus.cc:1212] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:04.351238 5235 raft_consensus.cc:1212] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20250110 03:08:04.438215 6553 raft_consensus.cc:1059] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER e8ba4f91a2c94c3883d60f7002e12bb2 to VOTER
I20250110 03:08:04.438403 6553 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } }
I20250110 03:08:04.438916 4513 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:04.438971 5235 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:04.438970 4381 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:04.439148 6710 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:04.439252 6553 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:04.439352 6629 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:04.440069 6710 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.440199 4513 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.440227 5235 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.441310 4233 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 8 to 9, e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:04.441772 4381 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.442876 4646 consensus_queue.cc:237] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } }
I20250110 03:08:04.443300 4513 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:04.443356 5235 raft_consensus.cc:1270] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:04.443550 6629 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:04.443648 6553 consensus_queue.cc:1035] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:04.444265 6710 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.444963 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a99a80262a3d492d838cc0e0fc757ffc with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:04.445101 5235 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.445112 4236 catalog_manager.cc:5526] T a99a80262a3d492d838cc0e0fc757ffc P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:04.445133 4513 raft_consensus.cc:2949] T a99a80262a3d492d838cc0e0fc757ffc P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } attrs { promote: false } } }
I20250110 03:08:04.446327 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet a99a80262a3d492d838cc0e0fc757ffc with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:04.446581 6714 tablet_replica.cc:331] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:04.446672 6714 raft_consensus.cc:2238] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:04.446767 6714 raft_consensus.cc:2267] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:04.446981 6714 ts_tablet_manager.cc:1905] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:04.448129 6714 ts_tablet_manager.cc:1918] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:08:04.448175 6714 log.cc:1198] T a99a80262a3d492d838cc0e0fc757ffc P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/a99a80262a3d492d838cc0e0fc757ffc
I20250110 03:08:04.448503 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet a99a80262a3d492d838cc0e0fc757ffc (table rebalance_test_table_1 [id=c71b7e56bb624813b8b09afd7ef1ff7b]) successfully deleted
I20250110 03:08:04.577814 6715 ts_tablet_manager.cc:927] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: Initiating tablet copy from peer e8ba4f91a2c94c3883d60f7002e12bb2 (127.14.19.130:43399)
I20250110 03:08:04.578023 6715 tablet_copy_client.cc:323] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.130:43399
I20250110 03:08:04.578276 4534 tablet_copy_service.cc:140] P e8ba4f91a2c94c3883d60f7002e12bb2: Received BeginTabletCopySession request for tablet 56330dd8a2ef4597be2d809c1cf02000 from peer 8cb2e4afb3384e40879d605ea471b039 ({username='slave'} at 127.14.19.133:39839)
I20250110 03:08:04.578336 4534 tablet_copy_service.cc:161] P e8ba4f91a2c94c3883d60f7002e12bb2: Beginning new tablet copy session on tablet 56330dd8a2ef4597be2d809c1cf02000 from peer 8cb2e4afb3384e40879d605ea471b039 at {username='slave'} at 127.14.19.133:39839: session id = 8cb2e4afb3384e40879d605ea471b039-56330dd8a2ef4597be2d809c1cf02000
I20250110 03:08:04.578774 4534 tablet_copy_source_session.cc:215] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:04.579124 6715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56330dd8a2ef4597be2d809c1cf02000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:04.580173 6715 tablet_copy_client.cc:806] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:04.580313 6715 tablet_copy_client.cc:670] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:04.580888 6715 tablet_copy_client.cc:538] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:04.581439 6715 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap starting.
I20250110 03:08:04.584663 6715 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:04.584851 6715 tablet_bootstrap.cc:492] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: Bootstrap complete.
I20250110 03:08:04.584959 6715 ts_tablet_manager.cc:1397] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250110 03:08:04.585070 6715 raft_consensus.cc:357] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:04.585143 6715 raft_consensus.cc:738] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8cb2e4afb3384e40879d605ea471b039, State: Initialized, Role: LEARNER
I20250110 03:08:04.585172 6715 consensus_queue.cc:260] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: NON_VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: true } }
I20250110 03:08:04.585289 6715 ts_tablet_manager.cc:1428] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:08:04.585528 4534 tablet_copy_service.cc:342] P e8ba4f91a2c94c3883d60f7002e12bb2: Request end of tablet copy session 8cb2e4afb3384e40879d605ea471b039-56330dd8a2ef4597be2d809c1cf02000 received from {username='slave'} at 127.14.19.133:39839
I20250110 03:08:04.585588 4534 tablet_copy_service.cc:434] P e8ba4f91a2c94c3883d60f7002e12bb2: ending tablet copy session 8cb2e4afb3384e40879d605ea471b039-56330dd8a2ef4597be2d809c1cf02000 on tablet 56330dd8a2ef4597be2d809c1cf02000 with peer 8cb2e4afb3384e40879d605ea471b039
I20250110 03:08:04.742763 6666 raft_consensus.cc:1059] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:08:04.742939 6666 consensus_queue.cc:237] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:04.743395 5235 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:04.743453 4646 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:04.743453 4381 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250110 03:08:04.743659 6734 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:04.743849 6668 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:04.743958 6592 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 03:08:04.744982 6734 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:04.745123 4646 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:04.745189 5235 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:04.746310 4233 catalog_manager.cc:5526] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:04.746450 4381 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
W20250110 03:08:04.747391 6592 consensus_queue.cc:1159] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e3a520ce69d74074a3d309fd843bccd6" responder_term: 1 status { last_received { term: 1 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 6 } }
I20250110 03:08:04.747494 4513 consensus_queue.cc:237] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:04.747956 4646 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:04.747977 5235 raft_consensus.cc:1270] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20250110 03:08:04.748136 6592 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:04.748226 6734 consensus_queue.cc:1035] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 03:08:04.748775 6592 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:04.748888 4646 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P b680ae46493d4601a57d0545c65a1895 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:04.748909 5235 raft_consensus.cc:2949] T 5f78c4d4f4de4b83b2b63a3b31e20fae P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:04.749608 4224 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:04.749775 4240 catalog_manager.cc:5526] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 6 to 7, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:04.750592 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f78c4d4f4de4b83b2b63a3b31e20fae with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:04.750720 6714 tablet_replica.cc:331] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:04.750788 6714 raft_consensus.cc:2238] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:04.750866 6714 raft_consensus.cc:2267] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:04.751111 6714 ts_tablet_manager.cc:1905] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:04.752167 6714 ts_tablet_manager.cc:1918] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20250110 03:08:04.752213 6714 log.cc:1198] T 5f78c4d4f4de4b83b2b63a3b31e20fae P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f78c4d4f4de4b83b2b63a3b31e20fae
I20250110 03:08:04.752547 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet 5f78c4d4f4de4b83b2b63a3b31e20fae (table rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]) successfully deleted
I20250110 03:08:05.065346 5235 raft_consensus.cc:1212] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:05.631529 6734 raft_consensus.cc:1059] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2: attempting to promote NON_VOTER 8cb2e4afb3384e40879d605ea471b039 to VOTER
I20250110 03:08:05.631716 6734 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:05.632234 5235 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 LEARNER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:05.632270 5365 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:05.632280 4381 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:05.632432 6766 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:05.632540 6765 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:05.632745 6763 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:05.633347 6766 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.633599 5365 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.633752 4381 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.634912 4236 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:05.634949 5235 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e3a520ce69d74074a3d309fd843bccd6" member_type: VOTER last_known_addr { host: "127.14.19.129" port: 37651 } attrs { replace: true } } peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.636117 4513 consensus_queue.cc:237] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } }
I20250110 03:08:05.636627 5365 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:05.636688 5235 raft_consensus.cc:1270] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer e8ba4f91a2c94c3883d60f7002e12bb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:05.636818 6765 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:05.636919 6763 consensus_queue.cc:1035] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:05.637483 6763 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.637602 5365 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.638514 4224 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 56330dd8a2ef4597be2d809c1cf02000 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:05.638635 4240 catalog_manager.cc:5526] T 56330dd8a2ef4597be2d809c1cf02000 P 22901baa49564439bfd6978fe2e4a4a9 reported cstate change: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New cstate: current_term: 1 leader_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.638998 5235 raft_consensus.cc:2949] T 56330dd8a2ef4597be2d809c1cf02000 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false } } }
I20250110 03:08:05.639860 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 56330dd8a2ef4597be2d809c1cf02000 with delete_type TABLET_DATA_TOMBSTONED (TS e3a520ce69d74074a3d309fd843bccd6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53980
I20250110 03:08:05.640075 6775 tablet_replica.cc:331] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: stopping tablet replica
I20250110 03:08:05.640144 6775 raft_consensus.cc:2238] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:05.640228 6775 raft_consensus.cc:2267] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:05.640448 6775 ts_tablet_manager.cc:1905] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:05.641598 6775 ts_tablet_manager.cc:1918] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:08:05.641649 6775 log.cc:1198] T 56330dd8a2ef4597be2d809c1cf02000 P e3a520ce69d74074a3d309fd843bccd6: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-0/wal/wals/56330dd8a2ef4597be2d809c1cf02000
I20250110 03:08:05.641961 4224 catalog_manager.cc:4872] TS e3a520ce69d74074a3d309fd843bccd6 (127.14.19.129:37651): tablet 56330dd8a2ef4597be2d809c1cf02000 (table rebalance_test_table_0 [id=b6e5fdbea3ff41c88050be5fccfd8168]) successfully deleted
I20250110 03:08:05.892203 5346 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:08:05.892957 4361 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:05.893932 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:08:05.899315 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:08:05.901264 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:05.936162 5346 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:08:05.937252 4361 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:05.937711 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:08:05.938333 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:05.938411 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:08:05.974721 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:05.976473 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:08:05.979151 4361 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:05.980432 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:08:05.982766 5346 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:08:06.017443 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:06.018126 5346 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:08:06.018625 4361 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:06.021548 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:08:06.021865 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:08:06.033492 4646 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:08:06.034001 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20250110 03:08:06.034009 5235 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20250110 03:08:06.034215 4580 consensus_peers.cc:487] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 -> Peer 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132:41431): Couldn't send request to peer 22901baa49564439bfd6978fe2e4a4a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b6206c7b3014154ba2689f574532355. This is attempt 1: this message will repeat every 5th retry.
I20250110 03:08:06.034221 6629 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:06.034355 6923 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250110 03:08:06.035076 6922 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:06.035276 5235 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:06.035283 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:06.036345 4240 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 reported cstate change: config changed from index 7 to 8, NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) added. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } } }
I20250110 03:08:06.501024 6941 ts_tablet_manager.cc:927] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: Initiating tablet copy from peer b680ae46493d4601a57d0545c65a1895 (127.14.19.131:40061)
I20250110 03:08:06.501185 6941 tablet_copy_client.cc:323] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Beginning tablet copy session from remote peer at address 127.14.19.131:40061
I20250110 03:08:06.501410 4666 tablet_copy_service.cc:140] P b680ae46493d4601a57d0545c65a1895: Received BeginTabletCopySession request for tablet 8b6206c7b3014154ba2689f574532355 from peer 22901baa49564439bfd6978fe2e4a4a9 ({username='slave'} at 127.14.19.132:44479)
I20250110 03:08:06.501489 4666 tablet_copy_service.cc:161] P b680ae46493d4601a57d0545c65a1895: Beginning new tablet copy session on tablet 8b6206c7b3014154ba2689f574532355 from peer 22901baa49564439bfd6978fe2e4a4a9 at {username='slave'} at 127.14.19.132:44479: session id = 22901baa49564439bfd6978fe2e4a4a9-8b6206c7b3014154ba2689f574532355
I20250110 03:08:06.501914 4666 tablet_copy_source_session.cc:215] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: Tablet Copy: opened 0 blocks and 1 log segments
I20250110 03:08:06.502247 6941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6206c7b3014154ba2689f574532355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 03:08:06.503293 6941 tablet_copy_client.cc:806] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 0 data blocks...
I20250110 03:08:06.503422 6941 tablet_copy_client.cc:670] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Starting download of 1 WAL segments...
I20250110 03:08:06.503953 6941 tablet_copy_client.cc:538] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 03:08:06.504467 6941 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap starting.
I20250110 03:08:06.507668 6941 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 03:08:06.507856 6941 tablet_bootstrap.cc:492] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: Bootstrap complete.
I20250110 03:08:06.507951 6941 ts_tablet_manager.cc:1397] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250110 03:08:06.508055 6941 raft_consensus.cc:357] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:08:06.508121 6941 raft_consensus.cc:738] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22901baa49564439bfd6978fe2e4a4a9, State: Initialized, Role: LEARNER
I20250110 03:08:06.508147 6941 consensus_queue.cc:260] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: NON_VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: true } }
I20250110 03:08:06.508237 6941 ts_tablet_manager.cc:1428] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20250110 03:08:06.508409 4666 tablet_copy_service.cc:342] P b680ae46493d4601a57d0545c65a1895: Request end of tablet copy session 22901baa49564439bfd6978fe2e4a4a9-8b6206c7b3014154ba2689f574532355 received from {username='slave'} at 127.14.19.132:44479
I20250110 03:08:06.508458 4666 tablet_copy_service.cc:434] P b680ae46493d4601a57d0545c65a1895: ending tablet copy session 22901baa49564439bfd6978fe2e4a4a9-8b6206c7b3014154ba2689f574532355 on tablet 8b6206c7b3014154ba2689f574532355 with peer 22901baa49564439bfd6978fe2e4a4a9
I20250110 03:08:07.049697 5365 raft_consensus.cc:1212] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20250110 03:08:07.461889 6629 raft_consensus.cc:1059] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895: attempting to promote NON_VOTER 22901baa49564439bfd6978fe2e4a4a9 to VOTER
I20250110 03:08:07.462085 6629 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:07.462682 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:07.462769 5235 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:07.462822 5365 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 LEARNER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20250110 03:08:07.463024 6966 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:07.463188 6967 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:07.463313 6629 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 03:08:07.464277 6967 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.464547 5235 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.464550 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.465659 4236 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250110 03:08:07.465745 5365 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 22901baa49564439bfd6978fe2e4a4a9 (127.14.19.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "8cb2e4afb3384e40879d605ea471b039" member_type: VOTER last_known_addr { host: "127.14.19.133" port: 34025 } attrs { promote: false replace: true } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.467901 4646 consensus_queue.cc:237] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } }
I20250110 03:08:07.468297 5365 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:07.468307 4513 raft_consensus.cc:1270] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Refusing update from remote peer b680ae46493d4601a57d0545c65a1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20250110 03:08:07.468483 6966 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:07.468567 6629 consensus_queue.cc:1035] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250110 03:08:07.469161 6629 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.469285 4513 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P e8ba4f91a2c94c3883d60f7002e12bb2 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.469385 5365 raft_consensus.cc:2949] T 8b6206c7b3014154ba2689f574532355 P 22901baa49564439bfd6978fe2e4a4a9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } } }
I20250110 03:08:07.470080 4225 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8b6206c7b3014154ba2689f574532355 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 03:08:07.470253 4240 catalog_manager.cc:5526] T 8b6206c7b3014154ba2689f574532355 P b680ae46493d4601a57d0545c65a1895 reported cstate change: config changed from index 9 to 10, VOTER 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133) evicted. New cstate: current_term: 1 leader_uuid: "b680ae46493d4601a57d0545c65a1895" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e8ba4f91a2c94c3883d60f7002e12bb2" member_type: VOTER last_known_addr { host: "127.14.19.130" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b680ae46493d4601a57d0545c65a1895" member_type: VOTER last_known_addr { host: "127.14.19.131" port: 40061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22901baa49564439bfd6978fe2e4a4a9" member_type: VOTER last_known_addr { host: "127.14.19.132" port: 41431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250110 03:08:07.473208 5215 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6206c7b3014154ba2689f574532355 with delete_type TABLET_DATA_TOMBSTONED (TS 8cb2e4afb3384e40879d605ea471b039 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33544
I20250110 03:08:07.473397 6983 tablet_replica.cc:331] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: stopping tablet replica
I20250110 03:08:07.473516 6983 raft_consensus.cc:2238] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 03:08:07.473590 6983 raft_consensus.cc:2267] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 03:08:07.473878 6983 ts_tablet_manager.cc:1905] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 03:08:07.474965 6983 ts_tablet_manager.cc:1918] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20250110 03:08:07.475018 6983 log.cc:1198] T 8b6206c7b3014154ba2689f574532355 P 8cb2e4afb3384e40879d605ea471b039: Deleting WAL directory at /tmp/dist-test-taskWQrDfu/test-tmp/rebalancer_tool-test.4.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1736478394679828-14414-0/raft_consensus-itest-cluster/ts-4/wal/wals/8b6206c7b3014154ba2689f574532355
I20250110 03:08:07.475328 4224 catalog_manager.cc:4872] TS 8cb2e4afb3384e40879d605ea471b039 (127.14.19.133:34025): tablet 8b6206c7b3014154ba2689f574532355 (table rebalance_test_table_2 [id=0dfe483bc8fd4d47bbec40378f5506c6]) successfully deleted
I20250110 03:08:07.694198 5346 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:08:07.694450 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:07.696035 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:08:07.698369 4361 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:07.703047 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:08:07.731611 5346 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 03:08:07.732342 5215 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:07.732498 4625 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20250110 03:08:07.735248 4492 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20250110 03:08:07.739991 4361 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 03:08:07.753266 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 4300
I20250110 03:08:07.758350 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 4432
I20250110 03:08:07.763371 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 4564
I20250110 03:08:07.768960 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 5284
I20250110 03:08:07.773736 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 5153
I20250110 03:08:07.778110 14414 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskWQrDfu/build/release/bin/kudu with pid 4209
2025-01-10T03:08:07Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (17160 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (28700 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (93119 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3, where GetParam() = { rep_factor: 3, num_tables: 3, location_info: { /A:3 /B:3 } }
1 FAILED TEST